Optimizing audio delivery for virtual reality applications

Abstract

Technologies, systems, methods, and instructions for virtual reality (VR), augmented reality (AR), mixed reality (MR), or 360-degree video environments are disclosed. In one example, a system (102) includes at least one media video decoder configured to decode a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to a user. The system includes at least one audio decoder (104) configured to decode an audio signal (108) from at least one audio stream (106). The system (102) is configured to request (112) at least one audio stream (106) and / or an audio element of an audio stream and / or an adaptation set from a server (120) based at least on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110).

Description

[0001] This application is a divisional application of the PCT international application PCT / EP2018 / 077770 with an application date of October 11, 2018, which was submitted to the China Patent Office on June 11, 2020 and entered the Chinese national phase, and the corresponding invention patent application No. 201880080196.6 entitled "Audio Delivery Optimization for Virtual Reality Applications". Background Art

[0002] illustrate

[0003] introduce

[0004] In a virtual reality (VR) environment or similarly in an augmented reality (AR) or mixed reality (MR) or 360-degree video environment, a user can typically use, for example, a head-mounted display (HMD) to visualize the full 360-degree content and listen through headphones (or similarly through loudspeakers, including correct rendering depending on their position).

[0005] In a simple use case, the content is licensed in such a way that only one audio / video scene (e.g., a 360-degree video) is reproduced at a specific point in time. The audio / video scene has a fixed position (e.g., a sphere with the user at the center), and the user may not move in the scene, but only rotate their head in various directions (yaw, pitch, roll). In this case, different video and audio are played to the user (displaying different viewports) based on the orientation of the user's head.

[0006] However, for video, the video content is transmitted for the entire 360-degree scene along with metadata describing the rendering process (e.g., stitching information, projection mapping, etc.), and is selected based on the current user's viewport. For audio, the content is the same for the entire scene. Based on the metadata, the audio content is adapted to the current user's viewport (e.g., audio objects are rendered differently based on viewport / user orientation information). It should be noted that 360-degree content refers to any type of content that includes more than one perspective at the same moment and that a user can select from (e.g., by user head orientation or by using a remote control).

[0007] In more complex scenarios, the audio content may also change as the user moves through the VR scene or "jumps" from one scene to the next (for example, an audio source that is inaudible in one scene may be audible in the next - a "door opening"). Using existing systems, the complete audio scene can be encoded as one stream, and if needed, additional streams (depending on the main stream). Such systems are known as next-generation audio systems (e.g., MPEG-H 3D Audio). Examples of such use cases may include:

[0008] Example 1: User chooses to enter a new room and the entire audio / video scene changes

[0009] Example 2: The user moves through the VR scene, opens a door, and walks through it, meaning the audio needs to transition from one scene to the next

[0010] To describe this scenario, the concept of discrete viewpoints in space is introduced, as discrete locations in space (or in a VR environment) for which different audio / video content is available.

[0011] A "direct" approach would have a real-time encoder that changes the encoding (number of audio elements, spatial information, etc.) based on feedback from the playback device about the user's position / orientation. This approach implies, for example, very complex communication between the client and the server in a streaming environment:

[0012] • The client (usually assumed to use only simple logic) will need advanced mechanisms to communicate not only requests for different streams, but also complex information about the encoding details that will enable processing of the correct content based on the user's location.

[0013] The media server is typically pre-populated with different streams (formatted in a specific way that allows for "segmented" delivery), and the server's main function is to provide information about available streams and cause their delivery when requested. To enable scenarios that allow encoding based on feedback from playback devices, the media server will need advanced communication links with multiple real-time media encoders, as well as the ability to create all signaling information (e.g., media presentation descriptions) on the fly, which may change in real time.

[0014] While such systems are conceivable, their complexity and computational requirements are beyond the capabilities and features of devices and systems available today, or even to be developed in the coming decades.

[0015] Alternatively, content representing the complete VR environment (the "full world") could be delivered all the time. This would solve the problem, but would require a huge bit rate that exceeds the capabilities of the available communication link.

[0016] For a real-time environment this is complex and in order to implement such a use case using available systems, alternatives are proposed to implement this functionality with less complexity.

[0017] 2. Terms and Definitions

[0018] The following terms are used in the technical field:

[0019] • Audio elements: audio signals that can be represented as, for example, audio objects, audio channels, scene-based audio (Higher Order Ambiance - HOA), or any combination of all.

[0020] Region of Interest (ROI): An area of ​​video content (or displayed or simulated environment) that the user is interested in at a given moment. This can typically be, for example, a region on a sphere, or a polygonal selection from a 2D map. The ROI identifies a specific area for a specific purpose, defining the boundaries of the object under consideration.

[0021] User location information: location information (e.g., x, y, z coordinates), orientation information (yaw, pitch, roll), and movement direction and speed.

[0022] Viewport: The portion of the spherical video that is currently displayed and viewed by the user.

[0023] Viewpoint: The center point of the viewport.

[0024] 360-degree video (also known as immersive video or spherical video): In the context of this document, this refers to video content that contains more than one view (i.e., viewport) in a single direction at the same moment in time. Such content can be created, for example, using an omnidirectional camera or a collection of cameras. During playback, the viewer can control the viewing direction.

[0025] • Media Presentation Description (MPD) is a syntax such as XML that contains information about media segments, the relationships between them, and the information necessary to select between them.

[0026] Adaptation sets contain media streams or sets of media streams. In the simplest case, one adaptation set contains all the audio and video for the content, but to reduce bandwidth, each stream can be split into a different adaptation set. A common case is to have one video adaptation set and multiple audio adaptation sets (one for each supported language). Adaptation sets can also contain subtitles or arbitrary metadata.

[0027] Representations allow adaptation sets to contain the same content encoded in different ways. In most cases, representations will be provided at multiple bitrates. This allows clients to request the highest quality content they can play without having to wait for buffering. Representations can also be encoded using different codecs, allowing support for clients with different supported codecs.

[0028] In the context of this application, the concept of adaptation set is used more generally, sometimes actually referring to representation. In addition, media streams (audio / video streams) are usually first encapsulated into media segments, which are the actual media files played by the client (e.g., DASH client). Various formats can be used for media segments, such as the ISO Base Media File Format (ISOBMFF) or MPEG-2 Transport Stream (TS) similar to the MPEG-4 container format. Encapsulation into media segments and encapsulation in different representations / adaptation sets are independent of the methods described in this article, which apply to all various options.

[0029] Furthermore, the description of the methods in this document is centered around DASH server-client communication, but the methods are general enough to be used with other delivery environments (e.g., MMT, MPEG-2TS, DASH-ROUTE, file formats for file playback, etc.).

[0030] Generally speaking, adaptation sets are at a higher level relative to streams and can include metadata (e.g., associated with positions). A stream can include multiple audio elements. An audio scene can be associated with multiple streams delivered as part of multiple adaptation sets.

[0031] 3 Current Plan

[0032] The current plan is:

[0033] [1].ISO / IEC 23008-3: 2015, Information technology--High efficiencycoding and media delivery in heterogeneous environments--Part 3: 3D audio

[0034] [2].N16950, Study of ISO / IEC DIS 23000-20 Omnidirectional MediaFormat.

[0035] Current solutions are limited to providing a standalone VR experience at one fixed location, which allows the user to change their orientation but not move within the VR environment. Summary of the Invention

[0036] According to an embodiment, a system for virtual reality VR, augmented reality AR, mixed reality MR or a 360-degree video environment may be configured to receive video and audio streams to be reproduced in a media consumption device, wherein the system may include: at least one media video decoder configured to decode a video signal from a video stream to represent a VR, AR, MR or 360-degree video environment scene to a user, and at least one audio decoder configured to decode an audio signal from at least one audio stream, wherein the system may be configured to: request at least one audio stream and / or an audio element of an audio stream and / or an adaptation set from a server based at least on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0037] According to one aspect, the system can be configured to provide the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data to the server to obtain at least one audio stream and / or an audio element of an audio stream and / or an adaptation set from the server.

[0038] Embodiments may be configured such that at least one scene is associated with at least one audio element, each audio element being associated with a location and / or area in the visual environment where the audio element is audible, such that different audio streams are provided for different users' positions and / or viewports and / or head orientations and / or movement data and / or interaction metadata and / or virtual position data in the scene.

[0039] According to another aspect, the system may be configured to decide whether to reproduce at least one audio element and / or an adaptation set of an audio stream for a current user's viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position in a scene, and wherein the system may be configured to request and / or receive the at least one audio element at the current user's virtual position.

[0040] According to another aspect, the system may be configured to predictively decide whether at least one audio element of an audio stream and / or an adaptation set will become relevant and / or audible based at least on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and wherein the system may be configured to request and / or receive at least one audio element and / or audio stream and / or adaptation set at a specific user's virtual position prior to predicted movement and / or interaction of the user in the scene, and wherein the system may be configured to reproduce at least one audio element and / or audio stream at the specific user's virtual position upon receipt of the at least one audio element and / or voice stream after the user's movement and / or interaction in the scene.

[0041] Embodiments of the system may be configured to request and / or receive at least one audio element at a lower bit rate and / or quality level at the user's virtual location prior to the user's movement and / or interaction in the scene, wherein the system may be configured to request and / or receive at least one audio element at a higher bit rate and / or quality level at the user's virtual location after the user's movement and / or interaction in the scene.

[0042] According to one aspect, the system can be configured such that at least one audio element is associated with at least one scene, each audio element being associated with a location and / or region in a visual environment associated with the scene, wherein the system can be configured to request and / or receive streams at a higher bitrate and / or quality level for audio elements that are closer to the user than for audio elements that are farther away from the user.

[0043] According to one aspect, in a system, at least one audio element may be associated with at least one scene, and at least one audio element may be associated with a location and / or area in a visual environment associated with the scene, wherein the system may be configured to request different streams at different bit rates and / or quality levels for the audio elements based on the relevance and / or audibility level of the audio elements at each user's virtual location in the scene, wherein the system may be configured to request audio streams at higher bit rates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location, and / or request audio streams at lower bit rates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location.

[0044] In an embodiment, in the system, at least one audio element may be associated with a scene, each audio element being associated with a location and / or area in the visual environment associated with the scene, wherein the system may be configured to periodically send the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data to a server to: provide a higher bitrate and / or quality stream from the server for a first location, and provide a lower bitrate and / or quality stream from the server for a second location, wherein the first location is closer to the at least one audio element than the second location.

[0045] In an embodiment, in the system, multiple scenes may be defined for multiple visual environments, such as adjacent and / or neighboring environments, thereby providing a first stream associated with a first current scene, and in the event of a user transition to another second scene, providing both a stream associated with the first scene and a second stream associated with the second scene.

[0046] In an embodiment, in the system, multiple scenes may be defined for a first and second visual environment, the first and second environments being adjacent and / or neighboring environments, wherein, in a case where the user's position or virtual position is in a first environment associated with the first scene, a first stream associated with the first scene is provided from a server for reproducing the first scene, in a case where the user's position or virtual position is in a second environment associated with the second scene, a second stream associated with the second scene is provided from a server for reproducing the second scene, and in a case where the user's position or virtual position is in a transition position between the first scene and the second scene, both the first stream associated with the first scene and the second stream associated with the second scene are provided.

[0047] In an embodiment, in the system, multiple scenes can be defined for a first and second visual environment, the first and second visual environments being adjacent and / or neighboring environments, wherein the system is configured to request and / or receive a first stream associated with a first scene associated with the first environment for reproducing the first scene when the user's virtual position is in the first environment, wherein the system can be configured to request and / or receive a second stream associated with a second scene associated with the second environment for reproducing the second scene when the user's virtual position is in the second environment, and wherein the system can be configured to request and / or receive both the first stream associated with the first scene and the second stream associated with the second scene when the user's virtual position is in a transition position between the first environment and the second environment.

[0048] According to one aspect, the system can be configured such that when the user is in a first environment associated with the first scene, a first stream associated with the first scene is obtained at a higher bit rate and / or quality, and when the user is at the beginning of a transition position from the first scene to the second scene, a second stream associated with the second scene associated with the second environment is obtained at a lower bit rate and / or quality, and when the user is at the end of the transition position from the first scene to the second scene, the first stream associated with the first scene is obtained at a lower bit rate and / or quality, and the second stream associated with the second scene is obtained at a higher bit rate and / or quality, wherein the lower bit rate and / or quality is lower than the higher bit rate and / or quality.

[0049] According to one aspect, the system can be configured such that multiple scenes are defined for multiple environments (e.g., adjacent and / or neighboring environments), such that the system can be configured to obtain a stream associated with a first current scene associated with a first current environment, and when the distance between the user's position or virtual position and the boundary of the scene is below a predetermined threshold, the system can also obtain an audio stream associated with a second adjacent and / or neighboring environment associated with a second scene.

[0050] According to one aspect, the system can be configured such that multiple scenes can be defined for multiple visual environments, so that the system requests and / or obtains a stream associated with the current scene at a higher bitrate and / or quality and requests and / or obtains a stream associated with a second scene at a lower bitrate and / or quality, where the lower bitrate and / or quality is lower than the higher bitrate and / or quality.

[0051] According to one aspect, the system can be configured such that multiple N audio elements can be defined, and when the distance of the user to the position or region of these audio elements is greater than a predetermined threshold, the N audio elements are processed to obtain a smaller number M of audio elements (M < N) associated with the position or region close to the position or region of the N audio elements, so as to provide at least one audio stream associated with the N audio elements to the system when the distance of the user to the position or region of the N audio elements is less than the predetermined threshold, or provide at least one audio stream associated with the M audio elements to the system when the distance of the user to the position or region of the N audio elements is greater than the predetermined threshold.

[0052] According to one aspect, the system can be configured such that at least one visual environment scene is associated with at least one of multiple N audio elements (N >= 2), each audio element being associated with a position and / or region in the visual environment, where at least one of the multiple N audio elements is provided in at least one representation at a high bitrate and / or quality level, and where at least one of the multiple N audio elements is provided in at least one representation at a low bitrate and / or quality level, where at least one representation is obtained by processing the N audio elements to obtain a smaller number M of audio elements (M < N) associated with the position or region close to the position or region of the N audio elements, where the system can be configured to request a representation at a higher bitrate and / or quality level for the audio elements when the audio elements are more relevant and / or more audible at the virtual position of the current user in the scene, and where the system can be configured to request a representation at a lower bitrate and / or quality level for the audio elements when the audio elements are less relevant and / or less audible at the virtual position of the current user in the scene.

[0053] According to one aspect, the system can be configured such that different streams are obtained for different audio elements when the distance and / or relevance and / or audibility level and / or angular orientation of the user is below a predetermined threshold.

[0054] In an embodiment, the system can be configured to request and / or obtain a stream based on the orientation of the user and / or the direction of movement of the user and / or user interaction in the scene.

[0055] In an embodiment of the system, a viewport may be associated with a position and / or a virtual position and / or movement data and / or a head.

[0056] According to one aspect, the system can be configured such that different audio elements are provided at different viewports, wherein the system can be configured to request and / or receive a first audio element at a higher bit rate if the first audio element falls within the viewport than a second audio element that does not fall within the viewport.

[0057] According to one aspect, a system can be configured to request and / or receive a first audio stream and a second audio stream, wherein a first audio element in the first audio stream is more relevant and / or more audible than a second audio element in the second audio stream, wherein the first audio stream is requested and / or received at a higher bitrate and / or quality than the bitrate and / or quality of the second audio stream.

[0058] According to one aspect, the system can be configured such that at least two visual environment scenes are defined, wherein at least one first and second audio elements are associated with a first scene associated with the first visual environment, and at least one third audio element is associated with a second scene associated with the second visual environment, wherein the system can be configured to obtain metadata describing that at least one second audio element is additionally associated with the second visual environment scene, and wherein, in a case where the user's virtual position is in the first visual environment, the system can be configured to request and / or receive at least the first and second audio elements, and wherein, in a case where the user's virtual position is in the second visual environment scene, the system can be configured to request and / or receive at least the second and third audio elements, and wherein, in a case where the user's virtual position is in a transition between the first visual environment scene and the second visual environment scene, the system can be configured to request and / or receive at least the first, second and third audio elements.

[0059] In one embodiment, the system can be configured such that at least one first audio element is provided in at least one audio stream and / or adaptation set, and at least one second audio element is provided in at least one second audio stream and / or adaptation set, and at least one third audio element is provided in at least one third audio stream and / or adaptation set, and wherein at least the first visual environment scene is described by metadata as a complete scene requiring at least the first and second audio streams and / or adaptation sets, and wherein the second visual environment scene is described by metadata as an incomplete scene requiring at least the third audio stream and / or adaptation set and at least the second audio stream and / or adaptation set associated with at least the first visual environment scene, wherein the system includes a metadata processor configured to manipulate the metadata to allow the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment to be merged into a new single stream when the user's virtual position is in the second visual environment.

[0060] According to one aspect, the system includes a metadata processor configured to manipulate metadata in at least one audio stream prior to at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0061] According to one aspect, the metadata processor may be configured to enable and / or disable at least one audio element in at least one audio stream before at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, wherein the metadata processor may be configured to disable at least one audio element in at least one audio stream before at least one audio decoder if the system decides not to reproduce the audio element due to the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and wherein the metadata processor may be configured to enable at least one audio element in at least one audio stream before at least one audio decoder if the system decides to reproduce the audio element due to the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0062] According to one aspect, the system may be configured to disable decoding of audio elements selected based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position.

[0063] According to one aspect, the system may be configured to merge at least one first audio stream associated with a current audio scene to at least one stream associated with an adjacent, neighboring and / or future audio scene.

[0064] According to one aspect, the system may be configured to obtain and / or collect statistical or aggregated data about the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual location data to send a request associated with the statistical or aggregated data to a server.

[0065] According to one aspect, the system can be configured to deactivate decoding and / or rendering of at least one stream based on metadata associated with the at least one stream and based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data.

[0066] According to one aspect, the system can be configured to: manipulate metadata associated with a selected group of audio streams based at least on a user's current or estimated viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to: select and / or enable and / or activate audio elements for synthesizing an audio scene to be reproduced; and / or allow all selected audio streams to be merged into a single audio stream.

[0067] According to one aspect, the system can be configured to control requests for at least one stream to the server based on the distance of the user's location from boundaries of adjacent and / or neighboring environments associated with different scenarios or a metric associated with the user's location in the current environment or a prediction of a future environment.

[0068] According to one aspect of the system, for each audio element or audio object information is provided from a server system, wherein the information comprises descriptive information about the location where the sound scene or audio element is active.

[0069] According to one aspect, the system can be configured to select between reproducing one scene and compositing or mixing or multiplexing or superimposing or combining at least two scenes, the two scenes being associated with different adjacent and / or neighboring environments, based on the current or future or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user's selection.

[0070] According to one aspect, the system can be configured to at least create or use adaptation sets such that: multiple adaptation sets are associated with one audio scene; and / or provide additional information associating each adaptation set with a viewpoint or an audio scene; and / or provide additional information that may include: information about the boundaries of an audio scene and / or information about the relationship between an adaptation set and an audio scene (for example, an audio scene is encoded in three streams, which are encapsulated in three adaptation sets) and / or information about the connection between the boundaries of the audio scene and multiple adaptation sets.

[0071] According to one aspect, the system can be configured to: receive a stream for a scene associated with an adjacent or proximate environment; and upon detecting a transition at a boundary between the two environments, begin decoding and / or rendering the stream for the adjacent or proximate environment.

[0072] According to one aspect, a system may be configured to include operating as a client and a server, the server being configured to deliver video and / or audio streams to be rendered in a media consumption device.

[0073] According to one aspect, the system can be configured to: request and / or receive at least one first adaptation set, the first adaptation set comprising at least one audio stream associated with at least one first audio scene; request and / or receive at least one second adaptation set, the second adaptation set comprising at least one second audio stream associated with at least two audio scenes, the at least two audio scenes including at least one first audio scene; and merge the at least one first audio stream and the at least one second audio stream into a new audio stream to be decoded based on available metadata related to the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or information describing the association of at least one first adaptation set with at least one first audio scene and / or the association of at least one second adaptation set with at least one first audio scene.

[0074] According to one aspect, the system can be configured to receive information about the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or any information characterizing changes triggered by the user's actions; and to receive information about the availability of adaptation sets and information describing the association of at least one adaptation set with at least one scene and / or viewpoint and / or viewport and / or position and / or virtual position and / or movement data and / or orientation.

[0075] According to one aspect, the system may be configured to: decide whether to reproduce at least one audio element from at least one audio scene embedded in at least one stream and at least one additional audio element from at least one additional audio scene embedded in at least one additional stream; and in case of a positive decision, cause an operation of merging or synthesizing or multiplexing or superimposing or combining at least one additional stream of the additional audio scene to at least one stream of the at least one audio scene.

[0076] According to one aspect, the system can be configured to: manipulate audio metadata associated with selected audio streams based at least on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to: select and / or enable and / or activate audio elements that synthesize the audio scene to be reproduced; and enable merging of all selected audio streams into a single audio stream.

[0077] According to one aspect, a server can be provided for delivering audio and video streams to a client for a virtual reality VR, augmented reality AR, mixed reality MR or 360-degree video environment, where the video and audio streams are to be reproduced in a media consumption device, wherein the server can include an encoder for encoding and / or a memory for storing a video stream describing a visual environment, the visual environment being associated with an audio scene; wherein the server can also include an encoder for encoding and / or a memory for storing a plurality of streams and / or audio elements and / or adaptation sets to be delivered to the client, the streams and / or audio elements and / or adaptation sets being associated with at least one audio scene, wherein the server is configured to: select and deliver a video stream based on a request from the client, the video stream being associated with the environment; select an audio stream and / or audio element and / or adaptation set based on a request from the client, the request being associated with at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data and with the audio scene associated with the environment; and deliver the audio stream to the client.

[0078] According to one aspect, streams may be encapsulated into adaptation sets, each adaptation set comprising multiple streams associated with different representations of the same audio content having different bitrates and / or qualities, wherein the selected adaptation set is selected based on a request from a client.

[0079] According to one aspect, the system can operate as both a client and a server.

[0080] According to one aspect, a system may include a server.

[0081] According to one aspect, a method for virtual reality (VR), augmented reality (AR), mixed reality (MR), or a 360-degree video environment may be provided, the method being configured to receive a video and / or audio stream to be reproduced in a media consumption device (e.g., a playback device), including: decoding a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to a user, and decoding an audio signal from an audio stream, wherein at least one audio stream is requested from and / or obtained from a server based on the user's current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata.

[0082] According to one aspect, a computer program may be provided, comprising instructions which, when executed by a processor, cause the processor to perform the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figures 1.1 to 1.8 An example of the present invention is shown.

[0084] Figures 2a to 6 A scenario illustrating the present invention.

[0085] Figures 7a to 8b The method of the present invention is shown. DETAILED DESCRIPTION

[0086] Aspects of the Invention

[0087] In the following (for example, Figures 1.1 to 1.8 ), which discloses an example of a system according to aspects of the present invention.

[0088] An example of a system of the present invention (which may be embodied by various examples disclosed below) is generally designated 102. System 102 may be, for example, a client system, as it may obtain audio and / or video streams from a server system (e.g., 120) to present an audio scene and / or visual environment to a user. Client system 102 may also receive metadata from server system 120 that provides, for example, secondary and / or auxiliary information about the audio and / or video streams.

[0089] System 102 can be associated with (or in some examples include) a media consumption device (MCD) that actually reproduces the audio and / or video signals to the user. In some examples, the user can wear the MCD.

[0090] The system 102 may perform a request to the server system 120 that is associated with the current viewport and / or head orientation (e.g., angular orientation) and / or movement data and / or interaction metadata and / or virtual position data 110 (which may provide several metrics) of at least one user. The viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 may be provided to the client system 102 in feedback from the MCD, and the client system 102 may, in turn, provide a request to the server system 120 based on the feedback.

[0091] In some cases, the request (indicated by 112) may include the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 (or an indication or processed version thereof). Based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110, the server system 120 will provide the required audio and / or video streams and / or metadata. In this case, the server system 120 may have knowledge of the user's location (e.g., within the virtual environment) and may associate the correct stream with the user's location.

[0092] In other cases, the request 112 from the client system 102 may include an explicit request for a specific audio and / or video stream. In this case, the request 112 may be based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location data 110. The client system 102 has knowledge of the audio and video signals that must be presented to the user, even if the client system 102 does not have the necessary streams stored therein. In an example, the client system 102 may address a specific stream in the server system 120.

[0093] The client system 102 may be a system for virtual reality (VR), augmented reality (AR), mixed reality (MR), or a 360-degree video environment configured to receive video and audio streams to be reproduced in a media consumption device.

[0094] The system 102 includes:

[0095] at least one media video decoder configured to decode a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to a user, and

[0096] at least one audio decoder 104 configured to decode an audio signal (108) from at least one audio stream 106,

[0097] The system 102 is configured to request 112 at least one audio stream 106 and / or an audio element of an audio stream and / or an adaptation set from a server 120 based at least on a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 of the user.

[0098] It should be noted that in VR, AR, and MR environments, user 140 may be said to be in a specific environment (e.g., a specific room). The environment is described using, for example, video signals encoded on the server side (server system 120 side, which need not necessarily include server system 120 but may include a different encoder that has previously encoded a video stream, which has subsequently been stored in the memory of server 120). In some examples, at any given moment, the user may only view some video signals (e.g., viewports).

[0099] Generally speaking, each environment can be associated with a specific audio scene. An audio scene can be understood as a collection of all sounds that will be reproduced to a user in a specific environment and within a specific time period.

[0100] Traditionally, environments have been understood as existing in a discrete number. Therefore, the number of environments has been understood to be finite. For the same reason, the number of audio scenes has been understood to be finite. Therefore, in the prior art, VR, AR, and MR systems have been designed so that:

[0101] -The user intends to be in a single environment at a time; therefore, for each environment:

[0102] o Client system 102 requests only video streams associated with a single environment from server system 120;

[0103] o The client system 102 only requests the audio stream associated with a single scene from the server system 120.

[0104] This method brings inconvenience.

[0105] For example, for each scene / environment, the audio streams would all be delivered together to the client system 102, and a completely new audio stream would need to be delivered when the user moves to a different environment (e.g., when the user passes through a door, thereby implying a transfer of environment / scene).

[0106] Furthermore, in some cases, this has led to an unnatural experience: for example, when a user is close to a wall (e.g., a virtual wall of a virtual room), they should experience the sound coming from the other side of the wall. However, this experience is impossible with traditional environments: the set of audio streams associated with the current scene obviously does not include any streams associated with adjacent environments / scenes.

[0107] On the other hand, increasing the bitrate of an audio stream generally improves the user experience. This can lead to other problems: the higher the bitrate, the higher the payload the server system needs to deliver to the client system 102. For example, when an audio scene contains multiple audio sources (transmitted as audio elements), some of which are located near the user's location and others farther away, the farther-away sound sources will be less audible. Therefore, transmitting all audio elements at the same bitrate or quality level may result in a very high bitrate. This means that ineffective audio streams are being delivered. If the server system 120 delivers the audio streams at the highest possible bitrate, ineffective delivery occurs because, similar to related sounds generated closer to the user, sounds with low audibility or low relevance to the overall audio scene still require a high bitrate. Therefore, if all audio streams of a scene are delivered at the highest bitrate, the communication payload between the server system 120 and the client system 102 will be unnecessarily increased. If all audio streams of a scene are delivered at a lower bitrate, the user experience will be unsatisfactory.

[0108] The problem of communication exacerbates the inconvenience discussed above: when a user passes through a door, it should be assumed that he changes the environment / scene immediately, which would require that the server system 120 should provide all streams to the client system 102 immediately.

[0109] Therefore, it has been conventionally impossible to solve the above problems.

[0110] However, with the present invention, these problems can be solved: the client system 102 provides a request to the server system 120, which can also be based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (and not only based on the environment / scene). Thus, the server system 120 can provide, for each instant, an audio stream to be rendered for each, for example, the user's position.

[0111] For example, if the user never approaches a wall, the client system 102 does not need to request streams for the adjacent environment (e.g., the client system 102 may request them only when the user approaches a wall). Additionally, the bitrate of streams from outside the wall may be reduced because they may be heard at a low volume. Notably, the server system 120 may transmit larger relevant streams (e.g., streams from audio objects within the current environment) to the client system 102 at the highest bitrate and / or highest quality level (due to the fact that smaller relevant streams are at lower bitrates and / or quality levels, thus leaving free bandwidth for larger relevant streams).

[0112] For example, a lower quality level can be achieved by reducing the bitrate or by processing audio elements in such a way that the required data to be transmitted is reduced while the used bitrate of each audio signal remains constant. For example, if 10 audio objects are located at different positions that are all far away from the user, these objects can be mixed into a lower number of signals based on the user's position:

[0113] - At locations far from the user's position (e.g. above a first threshold), the object is mixed into 2 signals (other numbers are possible based on its spatial location and semantics) and delivered as 2 "virtual objects"

[0114] - At locations closer to the user's location (e.g., below a first threshold but above a second threshold that is smaller than the first), objects are mixed into 5 signals (based on their spatial location and semantics) and delivered as 5 (other numbers are possible) "virtual objects"

[0115] At positions very close to the user's position (below the first and second thresholds), 10 objects are delivered as the 10 audio signals providing the highest quality.

[0116] When, at the highest quality, the audio signals are all considered significant and audible, the user can individually locate each object. At lower quality levels at distant locations, some audio objects may become less relevant or less audible, and the user will not be able to individually locate the audio signals in the space in any way, so reducing the quality level at which these audio signals are delivered will not result in any degradation in the user's quality of experience.

[0117] Another example is when a user passes through a door: in a transition position (e.g., at the boundary between two different environments / scenes), the server system 120 will provide two streams of both scenes / environments, but at a lower bitrate. This is because the user will experience sounds from two different environments (which may be merged together from different audio streams originally associated with different scenes / environments), so the highest quality level for each sound source (or audio element) is not required.

[0118] In view of the above, the present invention allows going beyond the traditional approach of a discrete number of visual environments and audio scenes, but may allow for a progressive representation of different environments / scenes, giving the user a more realistic experience.

[0119] In the following, each visual environment (e.g., a virtual environment) is considered to be associated with an audio scene (properties of the environment may also be properties of the scene). Each environment / scene may, for example, be associated with a geometric coordinate system (which may be a virtual geometric coordinate system). The environment / scene may have boundaries, so that when the user's position (e.g., virtual position) exceeds the boundaries, a different environment / scene is reached. The boundaries may be based on the coordinate system used. The environment may include audio objects (audio elements, sound sources) that can be positioned in certain specific coordinates of the environment / scene. Depending on, for example, the user's relative position and / or orientation with respect to the audio objects (audio elements, sound sources), the client system 102 may request different streams and / or the server system 120 may provide different streams (e.g., at a higher / lower bit rate and / or quality level depending on the distance and / or orientation).

[0120] More generally, client system 102 may request and / or obtain different streams (e.g., different representations of the same sound at different bit rates and / or quality levels) from server system 120 based on the audibility and / or relevance of the streams. Audibility and / or relevance may be determined, for example, based on at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location data.

[0121] In several examples, different streams can be merged. In several cases, there is the possibility of compositing or mixing or multiplexing or superimposing or combining at least two scenes. For example, there is the possibility of using a mixer and / or renderer (e.g., which can be used downstream of multiple decoders, each decoding at least one audio stream), or performing a stream multiplexing operation (e.g., upstream of the decoding of the streams). In other cases, there may be the possibility of decoding different streams and rendering them using different speaker setups.

[0122] It should be noted that the present invention does not necessarily exclude the concept of visual environments and audio scenes. In particular, with the present invention, when a user enters an environment / scene, the audio and video streams associated with a specific scene / environment can be delivered from the server system 120 to the client system 102. Nevertheless, in the same environment / scene, different audio streams and / or audio objects and / or adaptation sets can be requested, addressed and / or delivered. Specifically, the following possibilities may exist:

[0123] - at least some video data associated with the visual environment is communicated from the server 120 to the client 102 when the user enters the scene; and / or

[0124] - delivering at least some audio data (stream, object, adaptation set, ...) to the client system 102 based only on the current (or future) viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user's selection / interaction; and / or

[0125] - (In some cases): some audio data is delivered to the client system 102 based on the current scene (regardless of the current position or future position or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user's selection), while the rest of the audio data is delivered based on the current or future or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user's selection.

[0126] It is noted that the various elements (server system, client system, MCD, etc.) may represent elements in different hardware devices, or may even represent the same element (e.g., the client and MCD may be implemented as part of the same mobile phone, or similarly, the client may be on a PC connected to a secondary screen containing the MCD).

[0127] Example:

[0128] like Figure 1.1An embodiment of the system 102 (client) shown in FIG is configured to receive an (audio) stream 106 based on a defined position in an environment (e.g., a virtual environment), which environment can be understood as being associated with a video and audio scene (hereinafter referred to as scene 150). Typically, different positions in the same scene 150 mean different streams 106 or different metadata associated with the streams 106 to be provided to the audio decoder 104 of the system 102 (e.g., from a media server 120). The system 102 is connected to a media consumer device (MCD) from which it receives feedback associated with a user's position and / or virtual position in the same environment. In the following, the user's position in the environment can be associated with a specific viewport preferred by the user (viewport means, for example, the surface presented to the user, assumed to be a rectangular surface projected on a sphere).

[0129] In an exemplary scenario, as a user moves through a VR, AR, and / or MR scene 150, the audio content can be imagined as being virtually generated by one or more audio sources 152, which may change. Audio sources 152 can be understood as virtual audio sources, in the sense that they can refer to locations within the virtual environment: the rendering of each audio source is adapted to the user's location (e.g., in a simplified example, the level of the audio source is higher when the user is closer to the location of the audio source, and lower when the user is farther away). Nevertheless, each audio element (audio source) is encoded in an audio stream provided to the decoder. Audio streams can be associated with various locations and / or areas in the scene. For example, an audio source 152 that is inaudible in one scene can become audible in the next scene, e.g., when a door in the VR, AR, and / or MR scene 150 is opened. The user can then choose to enter a new scene / environment 150 (e.g., a room), and the entire audio scene changes. To describe this scenario, the term discrete viewpoints in space can be used as discrete locations in space (or in a VR environment) for which different audio content is available.

[0130] In general, the media server 120 can provide a stream 106 associated with a particular scene 150 based on the user's location in the scene 150. The stream 106 can be encoded by at least one encoder 154 and provided to the media server 120. The media server 120 can send the stream 113 via communication 113 (e.g., via a communication network). The provision of the stream 113 can be based on a request 112 made by the system 102 based on the user's location 110 (e.g., in the virtual environment). The user's location 110 can also be understood as being associated with the user's preferred viewport (for each location, there is a single rectangle represented) and viewpoint (e.g., the viewpoint is the center of the viewport). Therefore, in some examples, the provision of the viewport can be the same as the provision of the location.

[0131] like Figure 1.2 The system 102 shown is configured to receive (audio) streams 113 based on another configuration at the client side. In this example implementation, on the encoding side, a plurality of media encoders 154 are provided, which can be used to create one or more streams 106 for each available scene 150 associated with a sound scene portion of a viewpoint.

[0132] The media server 120 can store multiple audio and (not shown) video adaptation sets, which include different encodings of the same audio and video streams at different bit rates. In addition, the media server can include descriptive information for all adaptation sets, which can include the availability of all created adaptation sets. The adaptation sets can also include information describing the association of an adaptation set with a specific audio scene and / or viewpoint. In this way, each adaptation set can be associated with one of the available audio scenes.

[0133] Furthermore, the adaptation set may include information describing the boundaries of each audio scene and / or viewpoint, which may contain, for example, a complete audio scene or just individual audio objects. The boundaries of an audio scene may be defined as, for example, the geometric coordinates of a sphere (e.g., center and radius).

[0134] The client-side system 102 may receive information about the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or the user's virtual position, or any information characterizing changes triggered by the user's actions. In addition, the system 102 may also receive information about the availability of all adaptation sets and information describing the association of an adaptation set with an audio scene and / or viewpoint; and / or information describing the "boundaries" of each audio scene and / or viewpoint (which may contain, for example, a complete audio scene or only individual objects). For example, in the case of a DASH delivery environment, such information may be provided as part of the Media Presentation Description (MPD) XML syntax.

[0135] System 102 can provide audio signals to a media consumption device (MCD) for content consumption. The media consumption device is also responsible for collecting information about the user's position and / or orientation and / or direction of movement and / or the direction of movement (or information representing changes triggered by the user's actions) as position and translation data 110.

[0136] The viewport processor 1232 can be configured to receive the position and transformation data 110 from the media consumption device. The viewport processor 1232 can also receive information about the ROI signaled in the metadata and all information available at the receiving end (system 102). The viewport processor 1232 can then decide which audio viewpoint should be reproduced at a particular moment based on all the information received and / or derived from the received and / or available metadata. For example, the viewport processor 1232 can decide that a complete audio scene is to be reproduced and a new audio scene 108 must be created from all available audio scenes, for example, only some audio elements from multiple audio scenes are to be reproduced, while the remaining audio elements in these audio scenes are not reproduced. The viewport processor 1232 can also decide whether a transition between two or more audio scenes must be reproduced.

[0137] A selection section 1230 may be provided to select one or more adaptation sets signaled in the information received by the receiving end from the available adaptation sets based on the information received from the viewport processor 1232; the selected adaptation set completely describes the audio scene that should be reproduced at the user's current position. This audio scene may be a complete audio scene defined at the encoding side, or a new audio scene may have to be created from all available audio scenes.

[0138] Additionally, in the event that a transition between two or more audio scenes is about to occur based on an indication from the viewport processor 1232, the selection portion may be configured to select one or more adaptation sets signaled in the information received by the receiving end from the available adaptation sets; the selected adaptation set fully describes the audio scene that may need to be reproduced in the near future (e.g., if the user walks in the direction of the next audio scene at a particular speed, it may be predicted that the next audio scene will be necessary and that audio scene may be selected prior to reproduction).

[0139] Additionally, some adaptation sets corresponding to adjacent positions may be first selected at a lower bitrate and / or lower quality level, e.g., selecting representations encoded at a lower bitrate from the available representations in an adaptation set, and based on the position change, the quality may be improved by selecting a higher bitrate for those particular adaptation sets, e.g., selecting representations encoded at a higher bitrate from the available representations in an adaptation set.

[0140] A downloading and switching portion 1234 may be provided to request one or more adaptation sets from the available adaptation sets from the media server based on an indication received from the selection portion, the downloading and switching portion 1234 being configured to receive one or more adaptation sets from the available adaptation sets from the media server and extract metadata information from all received audio streams.

[0141] A metadata processor 1236 can be provided to receive information about the received audio streams from the downloads and switches, which information can include audio metadata corresponding to each received audio stream. The metadata processor 1236 can also be configured to process and manipulate the audio metadata associated with each audio stream 113 based on information received from the viewport processor 1232, which can include information about the user's location and / or orientation and / or direction of movement 110, to select / enable the desired audio elements 152 for synthesizing the new audio scene indicated by the viewport processor 1232, allowing all audio streams 113 to be combined into a single audio stream 106.

[0142] The stream multiplexer / merger 1238 can be configured to combine all selected audio streams into one audio stream 106 based on information received from the metadata processor 1236, which information can include the modified and processed audio metadata corresponding to all received audio streams 113.

[0143] The media decoder 104 is configured to receive and decode at least one audio stream based on information about the user's location and / or orientation and / or direction of movement to reproduce the new audio scene indicated by the viewport processor 1232.

[0144] In another embodiment, the system 102 as Figure 1.7 shown can be configured to receive audio streams 106 at different audio bitrates and / or quality levels. The hardware configuration of this embodiment is similar to Figure 1.2 that of. At least one visual environment scene 152 can be associated with at least one of a plurality of N audio elements (N≥2), each audio element being associated with a location and / or region in the visual environment. At least at least one of the plurality of N audio elements 152 is provided at a high bitrate and / or quality level in at least one representation, and at least at least one of the plurality of N audio elements 152 is provided at a low bitrate and / or quality level in at least one representation, wherein at least one representation is obtained by processing the N audio elements 152 to obtain a smaller number M of audio elements 152 (M<N) associated with the location or region close to the location or region of the N audio elements 152.

[0145] The processing of the N audio elements 152 may be, for example, a simple addition of the audio signals, or may be an effective downmix based on their spatial positions 110 or a rendering of audio signals using their spatial positions to new virtual positions located between the audio signals. In the event that the audio element is more relevant and / or more audible at the virtual position of the current user in the scene, the system may be configured to request representation at a higher bit rate and / or quality level for the audio element, wherein the system is configured to request representation at a lower bit rate and / or quality level for the audio element if the audio element is less relevant and / or less audible at the virtual position of the current user in the scene.

[0146] Figure 1.8 An example of a system (which may be system 102) is shown, which shows system 102 for virtual reality VR, augmented reality AR, mixed reality MR, or a 360-degree video environment, which is configured to receive a video stream 1800 and an audio stream 106 to reproduce them in a media consumption device.

[0147] Among other things, the system 102 may include:

[0148] At least one media video decoder 1804 configured to decode a video signal 1808 from the video stream 1800 to represent a VR, AR, MR, or 360-degree video environment to a user, and

[0149] At least one audio decoder 104 is configured to decode an audio signal 108 from at least one audio stream 106 .

[0150] The system 102 may be configured to request (112) from a server (e.g., 120) at least one audio stream 106 and / or an audio element of an audio stream and / or an adaptation set based at least on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data 110 (e.g., provided as feedback from a media consumption device 180).

[0151] System 102 can be used with Figures 1.1 to 1.7 The system 102 is the same as and / or obtains Figure 2a and Figure 2b scene.

[0152] The present example also relates to a method for a virtual reality (VR), augmented reality (AR), mixed reality (MR), or 360-degree video environment configured to receive video and audio streams to be reproduced in a media consumption device (e.g., a playback device), comprising:

[0153] Decoding a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to the user, and

[0154] Decode the audio signal from the audio stream,

[0155] At least one audio stream is requested from and / or obtained from the server based on the user's current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata.

[0156] Case 1

[0157] Typically, different scenes / environments 150 mean that different streams 106 are received from the server 120. However, the stream 106 received by the audio decoder 104 may also be adjusted by the position of the user in the same scene 150.

[0158] exist Figure 2a At a first (start) time (t=t1) shown in FIG, the user is positioned, for example, in a scene 150, with a first defined position in the VR environment (or AR environment or MR environment). In a Cartesian XYZ system (e.g., horizontal), the user's first viewport (position) 110′ is aligned with coordinate x′ u and y′ u (The Z axis is oriented out of the paper here.) In this first scene 150, two audio elements 152-1 and 152-1 are positioned, with corresponding coordinates x'1 and y'1 for audio element 1 (152-1), and corresponding coordinates x'2 and y'2 for audio element 2 (152-2). The distance d'1 from the user to audio element 1 (152-1) is less than the distance d'2 from the user to audio element 2 (152-1). All user position (viewport) data is sent from the MCD to the system 102.

[0159] exist Figure 2b At the second exemplary time (t=t2) shown in FIG, the user is positioned in, for example, the same scene 150, but in a different second position. In a Cartesian XY system, the user's second viewport (position) 110" corresponds to the new coordinate x" u and y" u (The Z axis here is oriented to exit from the paper.) Now, the user's distance d"1 from audio element 1 (152-1) is greater than the user's distance d"2 from audio element 2 (152-2). All user position (viewport) data is again sent from the MCD to the system 102.

[0160] A user equipped with the MCD for visualizing a certain viewport within a 360 degree environment may be listening, for example, through headphones. The user may enjoy a video of the same scene 150. Figure 2a and Figure 2b The reproduction of different sounds in different locations described in.

[0161] For example, Figure 2a and Figure 2b Any position within the scene and / or any transformation and / or viewport and / or virtual position and / or head orientation and / or movement data is periodically sent (e.g., in a feedback manner) from the MCD to the system 102 (client) as a signal 110. The client may resend the position and transformation data 110′ or 110″ (e.g., viewport data) to the server 120. The client 102 or the server 120 may decide, based on the position and transformation data 110′ or 110″ (e.g., viewport data), which audio streams 106 are needed to reproduce the correct audio scene at the current user position. The client may decide and send a request 112 for the corresponding audio stream 106, and the server 120 may be configured to deliver the stream 106 accordingly based on the position information provided by the client (system 102). Alternatively, the server 120 may decide and deliver the stream 106 accordingly based on the position information provided by the client (system 102).

[0162] The client (system 102) may request that a stream be sent that is to be decoded to represent a scene 150. In some examples, the system 102 may send information regarding the highest quality level to be reproduced on the MCD (in other examples, the server 120 decides the quality level to be reproduced on the MCD based on the user's position in the scene). In response, the server 120 may select one of a plurality of representations associated with the audio scene to be represented to deliver at least one stream 106 according to the user's position 110' or 110". The client (system 102) may thus be configured to deliver the audio signal 108 to the user, e.g., via the audio decoder 104, to reproduce the sound associated with his actual (effective) position 110' or 110". (An adaptation set 113 may be used: different variants of the same stream (e.g., at different bit rates) may be used for different positions of the user.)

[0163] The stream 106 (which may be pre-processed or generated on the fly) may be sent to the client (system 102) and may be configured for multiple viewpoints associated with a particular sound scene.

[0164] It has been noted that different qualities (e.g., different bit rates) may be provided for different streams 106 depending on the user's particular position (e.g., 110′ or 110″) in the (e.g., virtual) environment. For example, in the case of multiple audio sources 152-1 and 152-2, each audio source 152-1 and 152-2 may be associated with a particular position within the scene 150. The closer the user's position 110′ or 110′ is to the first audio source 152-1, the higher the required resolution and / or quality of the stream associated with the first audio source 152-2. This exemplary scenario may be applied to Figure 2a Audio element 1 (152-1) and Figure 2b The further away the user's location 110 is from the second audio source 152-2, the lower the resolution required for the stream 106 associated with the second audio source 152-2. This exemplary scenario may be applied to Figure 2a Audio element 2 (152-2) and Figure 2b Audio element 1 (152-1) in.

[0165] In effect, first, closer audio sources will be heard at a higher level (and therefore provided at a higher bitrate), while second, more distant audio sources will be heard at a lower level (possibly allowing for the need for a lower resolution);

[0166] Thus, based on the location 110′ or 110″ in the environment provided by the client 102, the server 120 can provide different streams 106 at different bit rates (or other qualities). Based on the fact that distant audio elements do not require a high level of quality, the overall quality of the user's experience can be maintained even if the audio is delivered at a lower bit rate or quality level.

[0167] Therefore, different quality levels can be used for certain audio elements at different user locations while maintaining the quality of experience.

[0168] Without this solution, the server 120 should provide all streams 106 to the client at the highest bit rate, which would increase the payload in the communication channel from the server 120 to the client.

[0169] Case 2

[0170] Figure 3 (Case 2) shows an embodiment with another exemplary scene (represented in a vertical plane XZ of the space XYZ, where the axis Y is represented as entering the paper) where a user moves in a first VR, AR and / or MR scene A (150A), opens a door and walks through (transition 150AB), which means that the audio transitions from the first scene 150A at time t1 through a temporary position (150AB) at time t2 to the next (second) scene B (150B) at time t3.

[0171] At time t1, the user may be at position x1 in the x-direction of a first VR, AR, and / or MR scene. At time t3, the user may be at a different second VR, AR, and / or MR scene B (150B) at position x3. At time t2, the user may be at transition position 150AB as they are opening a door (e.g., a virtual door) and walking through it. Thus, a transition means a transition of audio information from the first scene 150A to the second scene 150B.

[0172] In this context, the user is changing his position 110, for example, from the first VR environment (defined by Figure 1.1 The first viewpoint (A) shown in FIG) changes to the second VR environment (represented by Figure 1.1 In certain cases, for example, during a transition through a door located at position x2 in the x-direction, some audio elements 152A and 152B may be present in both viewpoints (positions A and B).

[0173] The user (equipped with an MCD) is changing his position 110 (x1-x3) toward the door, which may mean that in the transition position x2, the audio elements belong to both the first scene 150A and the second scene 150B. The MCD sends the new position and transition data 110 to the client, which resends it to the media server 120. The user can be enabled to listen to the appropriate audio source defined by the intermediate position x2 between the first position x1 and the second position x3.

[0174] Any position and any transformation from the first position (x1) to the second position (x3) are now periodically (e.g., continuously) sent from the MCD to the client. The client 102 can resend the position and transformation data 110 (x1-x3) to the media server 120, which is configured to deliver a dedicated item of the new pre-processed stream 106, for example in the form of an implemented adaptation set 113', based on the received position and transformation data 110 (x1-x3).

[0175] The media server 120 can select one of a plurality of representations associated with the aforementioned information, not only in relation to the MCD's ability to display the highest bitrate, but also in relation to the user's location and transition data 110 (x1-x3) during their movement from one location to another. (In this context, adaptation sets can be used: the media server 120 can decide which adaptation set 113′ best represents the user's virtual transition without interfering with the MCD's rendering capabilities.)

[0176] Thus, the media server 120 may deliver the dedicated stream 106 according to the switch of locations (eg, as a new adaptation set 113 ′). The client 102 may be configured to deliver the audio signal 108 to the user 140 accordingly, eg, via the media audio decoder 104 .

[0177] The (on-the-fly generated and / or pre-processed) stream 106 may be sent to the client 102 in an adaptation set 113 ′ that is implemented periodically (eg, continuously).

[0178] When the user walks through the door, the server 120 can send both the stream 106 of the first scene 150A and the stream 106 of the second scene 150B. This is to simultaneously mix, multiplex, synthesize, or reproduce these streams 106 to give the user a realistic impression. Therefore, based on the user's location 110 (e.g., "the location corresponding to the door"), the server 120 sends different streams 106 to the client.

[0179] Even in this case, since the different streams 106 will be heard simultaneously, they may have different resolutions and may be transmitted to the client at different resolutions from the server 120. When the user has completed the transition and is in the second (position) scene 150A (and has closed the door behind him), the server 120 will likely reduce or limit sending the stream 106 of the first scene 150 (if the server 120 has already provided streams to the client 102, the client 102 may decide not to use them).

[0180] Case 3

[0181] Figure 4 (Case 3) illustrates an embodiment with another exemplary scenario (represented in the vertical plane XZ of the space XYZ, with the axis Y represented as entering the paper) in which a user moves within the VR, AR, and / or MR scene 150A, meaning that the audio transitions from a first location at time t1 to a second location at time t2, also within the first scene 150A. The user in the first location may be far from the wall at time t1, at a distance d1 from the wall; and may be close to the wall at time t2, at a distance d2 from the wall. Here, d1 > d2. Although at distance d1, the user only hears source 152A of scene 150A, he may also hear source 152B of scene 150B beyond the wall.

[0182] When the user is at the second location (d2), the client 102 sends data related to the user's location 110 (d2) to the server 120, and receives not only the audio stream 106 of the first scene 150A but also the audio stream 106 of the second scene 150B from the server 120. For example, based on the metadata provided by the server 120, the client 102 will reproduce the stream 106 of the second scene 150B (outside the wall) at a low volume, for example, via the decoder 104.

[0183] Even in this case, the bitrate (quality) of the stream 106 of the second scene 150B can be lower, thus reducing the transmission payload from the server 120 to the client. It is worth noting that the position 110 (d1, d2) of the client (and / or viewport) defines the audio stream 106 provided by the server 120.

[0184] For example, the system 102 can be configured to obtain a stream associated with a first current scene (150A) that is associated with a first current environment and is captured when the distance between the user's position or virtual position and a boundary of the scene (e.g., corresponding to a wall) is below a predetermined threshold (e.g., when d2 < d threshold When , the system 102 also obtains an audio stream associated with a second adjacent and / or neighboring environment, the second adjacent and / or neighboring environment being associated with the second scene (150B).

[0185] Case 4

[0186] Figure 5a and Figure 5b An embodiment with another exemplary scene is shown (represented in a horizontal plane XY of a space XYZ, where axis Z is represented as exiting from the paper) in which a user is positioned in one and the same VR, AR and / or MR scene 150, but at different distances to, for example, two audio elements at different moments in time.

[0187] exist Figure 5a At a first time t=t1 shown in FIG, the user is positioned, for example, at a first position. At this first position, the first audio element 1 (152-1) and the second audio element 2 (152-2) are located at distances d1 and d2, respectively (e.g., physically) from the user equipped with the MCD. In this case, both distances d1 and d2 may be greater than a defined threshold distance d threshold , so the system 102 is configured to group the two audio elements into a single virtual source 152-3. The position and properties (e.g., spatial extent) of the single virtual source can be calculated based on, for example, the positions of the original two sources so that the single virtual source simulates the original sound field produced by the two sources as closely as possible (e.g., two well-positioned point sources can be reproduced as a single source at a distance intermediate between them). The user position data 110 (d1, d2) can be sent from the MCD to the system 102 (client) and then to the server 120, which can decide to send the appropriate audio stream 106 to be rendered by the server system 120 (in other embodiments, the client 102 decides which streams to send from the server 120). By grouping the two audio elements into a single virtual source 152-3, the server 120 can select one of a plurality of representations associated with the aforementioned information. (For example, dedicated stream 106 (adaptation set 113' correspondingly associated with, for example, a single channel) may be delivered accordingly.) Thus, a user may receive, via the MCD, an audio signal being transmitted from a single virtual audio element 152-3 located between real audio elements 1 (152-1) and 2 (152-2).

[0188] exist Figure 5b At the second time t=t2 shown, the user is positioned in, for example, the same scene 150 having Figure 5a A second position is defined in the same VR environment. At the second position, the two audio elements 152-1 and 152-2 are (eg, virtually) located at distances d3 and d4 from the user, respectively. Both distances d3 and d4 may be less than a threshold distance d threshold , thus grouping audio elements 152-1 and 152-2 into a single virtual source 152-3 is no longer used. User position data is sent from the MCD to the system 102 and subsequently to the server 120, which can decide to send another appropriate audio stream 106 to be rendered by the system server 120 (in other embodiments, this decision is made by the client 102). By avoiding grouping the audio elements, the server 120 can select a different representation associated with the aforementioned information to deliver a dedicated stream 106 with an adaptation set 113', which is correspondingly associated with a different channel of each audio element. Thus, the user can receive audio signals 108 sent from two different audio elements 1 (152-1) and 2 (152-2) through the MCD. Therefore, the closer the user's position 110 is to audio source 1 (152-1) and audio source 2 (152-2), the higher the required quality level of the stream associated with the audio source must be selected.

[0189] In fact, if Figure 5b As shown, the closer audio sources 1 (152-1) and 2 (152-2) are positioned relative to the user, the higher the level must be adjusted, and thus the audio signal 108 can be rendered at a higher quality level. Conversely, Figure 5b The remotely located audio sources 1 and 2 represented in must be listened to at a lower level than reproduced by a single virtual source and are therefore eg reproduced at a lower quality level.

[0190] In a similar configuration, multiple audio elements can be located in front of the user, all positioned at a distance greater than a threshold distance from the user. In one embodiment, each of the two groups of five audio elements can be combined in two virtual sources. The user position data is sent from the MCD to the system 102 and then to the server 120, which can decide to send the appropriate audio stream 106 to be rendered by the system server 120. By grouping all 10 audio elements into only two single virtual sources, the server 120 can select one of the multiple representations associated with the above information to deliver the dedicated stream 106 accordingly to the adaptation set 113', which is correspondingly associated with, for example, two single audio elements. Thus, the user can receive audio signals sent from two different virtual audio elements located in the same positioning area as the real audio element through the MCD.

[0191] At a subsequent moment, the user is approaching multiple (ten) audio elements. In this subsequent scenario, all audio elements are located less than the threshold distance d thresho1d distance, the system 102 is therefore configured to terminate the grouping of audio elements. The new user position data is sent from the MCD to the system 102 and subsequently to the server 120, which can decide to send another appropriate audio stream 106 for presentation by the server system 120. By not grouping the audio elements, the server 120 can select a different representation associated with the aforementioned information to deliver a dedicated stream 106 accordingly to the adaptation set 113', which is accordingly associated with a different channel of each audio element. Thus, the user can receive audio signals sent from ten different audio elements via the MCD. Therefore, the closer the user's position 110 is to the audio source, the higher the required resolution of the stream associated with the audio source that must be selected.

[0192] Case 5

[0193] Figure 6 (Scenario 5) depicts a user 140 in one location in a single scene 150 wearing a media consumer device (MCD) that can be directed in three exemplary different directions (each direction associated with a different viewport 160-1, 160-2, 160-3). Figure 6 The directions shown may have orientations (e.g., angular orientations) in a polar coordinate system and / or a Cartesian XY system that point, for example, to the Figure 6 A first viewpoint 801 positioned at 180° in the bottom of Figure 6 A second viewpoint 802 positioned at 90° on the right side of Figure 6 8. Each of these viewpoints is associated with an orientation of a user 140 wearing a media consumer device (MCD), with the user positioned in the center being provided with a specific viewport displayed by the MCD that renders a corresponding audio signal 108 depending on the orientation of the MCD.

[0194] In this particular VR environment, a first audio element s1 (152) is located in a first viewport 160-1 near a viewpoint located, for example, at 180°, and a second audio element s2 (152) is located in a third viewport 160-3 near a viewpoint located, for example, at 180°. Before changing his orientation, the user 140 experiences louder sound from audio element s1 than from audio element s2 in the first orientation toward viewpoint 801 (viewport 160-1), associated with his actual (effective) position.

[0195] By changing his orientation, the user 140 may experience sounds associated with his actual position 110 in a second orientation towards the viewpoint 802 , coming from the two audio elements s1 and s2 laterally at almost the same loudness.

[0196] Finally, by changing his orientation, user 140 can experience the sound associated with audio element 2 as louder than the sound associated with audio element s1 in a third orientation toward viewpoint 801 (viewport 160-3) (in effect, the sound from audio element 2 arrives from the front, while the sound from audio element 1 arrives from the back).

[0197] Thus, different viewports and / or orientations and / or virtual position data may be associated with different bit rates and / or qualities.

[0198] Other situations and examples

[0199] Figure 7a An embodiment of a method for receiving an audio stream by the system is shown in the form of a sequence of operational steps in the figure. At any time, a user of the system 102 is associated with his or her current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location. At a particular moment in time, Figure 7a In step 701, the system may determine the audio elements to be reproduced based on the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position. Thus, in the next step 703, the relevance and audibility of each audio element may be determined. Figure 6 As described in , a VR environment may have different audio elements not only located in a particular scene 150 near or further away from the user, but also at a particular orientation within the 360-degree environment. All of these factors determine the relevance and audibility level of each of the audio elements.

[0200] In a next step 705 , the system 102 may request an audio stream from the media server 120 based on the relevance and audibility levels determined for each audio element.

[0201] In a next step 707 , the system 102 may receive the audio streams 113 prepared accordingly by the media server 120 , wherein the streams with different bit rates may reflect the relevance and audibility levels as determined in the previous steps.

[0202] In a next step 709, the system 102 (e.g., an audio decoder) may decode the received audio stream 113 so that, at step 711, a particular scene 150 is reproduced (e.g., via an MCD) based on the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position.

[0203] Figure 7b The interaction between the media server 120 and the system 102 according to the aforementioned operational sequence diagram is depicted. At a particular moment, the media server may transmit the audio stream 750 at a lower bit rate based on the previously determined lower relevance and audibility level of the relevant audio elements of the aforementioned scene 150. The system may then determine at a subsequent moment 752 that an interaction or change in positional data has occurred. Such an interaction may be caused, for example, by a change in positional data from the same scene 150 or by a user activating a door handle when attempting to enter a second scene separate from the first scene through a door provided by the door handle.

[0204] A change in the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location may cause the system 102 to send a request 754 to the media server 120. The request may reflect the higher relevance and audibility level of the relevant audio elements determined for the subsequent scene 150. In response to the request 754, the media server may send a stream 756 at a higher bitrate, thereby enabling the system 102 to produce a reasonably realistic reproduction of the scene 150 at any current user's virtual location.

[0205] Figure 8a Another embodiment of a method for receiving an audio stream by a system is also shown in the form of a sequence of operational steps in the figure. At a specific time 801, a determination of a first current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position can be performed. By eliminating the positive case, at step 803, a request for a stream associated with the first position defined by the low bit rate can be prepared and sent by the system 102.

[0206] A determination step 805 with three different outcomes can be performed at a later time. One or two defined thresholds can be relevant in this step for determining a prediction decision, for example, related to the subsequent viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position. Thus, a comparison with the first and / or second thresholds can be performed regarding the probability of changing to the second position, resulting in, for example, three different subsequent steps to be performed.

[0207] In the event of a result reflecting, for example, a very low probability (eg, the probability associated with the above-mentioned comparison with the first predetermined threshold), a new comparison step 801 will be performed.

[0208] A result reflecting a low probability (eg, above a first predetermined threshold, but in an example, below a second predetermined threshold that is higher than the first threshold) may result in requesting the audio stream 113 at a low bit rate at step 809 .

[0209] Where a result reflects a high probability (eg above a second predetermined threshold), a request for the audio stream 113 at a high bit rate may be performed at step 807. The subsequent step to be performed after performing step 807 or 809 may therefore again be determining step 801.

[0210] Figure 8b The interaction between media server 120 and system 102 according to only one of the aforementioned operational sequence diagrams is depicted. At a particular moment, the media server may send an audio stream 850 at a low bitrate based on the previously determined low relevance and audibility levels of the audio elements of the aforementioned scene 150. The system may then determine at a subsequent moment 852 that an interaction will predictively occur. A predicted change in the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location may cause system 102 to send an appropriate request 854 to media server 120. This request may reflect one of the aforementioned conditions related to a high probability of reaching a second location associated with a high bitrate, based on the desired audibility levels of the audio elements for the corresponding subsequent scene 150. In response, the media server may send a stream 856 at a higher bitrate, enabling system 102 to provide a reasonably realistic reproduction of scene 150 at any current user's virtual location.

[0211] like Figure 1.3 The system 102 shown is configured to receive the audio stream 113 based on another configuration on the client side, where the system architecture can use discrete viewpoints based on a scheme using multiple audio decoders 1320, 1322. On the client side, the system 102 can be embodied as, for example Figure 1.2 Part of the system described in, which additionally or alternatively includes multiple audio decoders 1320, 1322, which can be configured to decode individual audio streams indicated by the metadata processor 1236 (e.g., where multiple audio elements are not activated).

[0212] A mixer / renderer 1238 may be provided in the system 102, which is configured to reproduce the final audio scene based on information about the user's position and / or orientation and / or movement direction, i.e., for example, some audio elements that are not audible at that particular position should be disabled or not rendered.

[0213] Figure 1.4 、 Figure 1.5 and Figure 1.6The following embodiments are based on independent adaptation sets for discrete viewpoints with flexible adaptation sets. As the user moves in the VR environment, the audio scene can change in a continuous manner. To ensure a good audio experience, all audio elements of the synthesized audio scene at a specific moment may have to be available to the media decoder, which can use the position information to create the final audio scene.

[0214] If the content is pre-encoded for multiple predefined locations, the system can provide accurate reproduction of the audio scenes at these specific locations, provided that these audio scenes do not overlap and the user can "jump / switch" from one location to the next.

[0215] However, in the case where a user "walks" from one location to the next, audio elements from two (or more) audio scenes may be heard simultaneously. A solution for this use case was provided in the previous system example, where, regardless of the mechanisms provided for decoding multiple audio streams (using a multiplexer with a single media decoder or multiple media decoders with an attached mixer / renderer), an audio stream describing the complete audio scene must be provided to the client.

[0216] The following provides an optimization by introducing the concept of common audio elements between multiple audio streams.

[0217] Discussion Aspects and Examples

[0218] Scheme 1: Independent adaptation sets for discrete locations (viewpoints).

[0219] One way to solve the above problem is to use a complete independent adaptation set for each position. To better understand the solution, Figure 1.1 Serves as an example scene. In this example, three different discrete viewpoints (including three different audio scenes) are used to create a complete VR environment that the user should be able to enter. Therefore:

[0220] · Multiple independent or overlapping audio scenes are encoded as multiple audio streams. For each audio scene, one main stream, one main stream and additional auxiliary streams can be used depending on the use case (for example, certain audio objects containing different languages ​​can be encoded as independent streams for efficient delivery). In the example provided, audio scene A is encoded as two streams (A1 and A2), audio scene B is encoded as three streams (B1, B2 and B3), and audio scene C is encoded as three streams (C1, C2 and C3). It should be noted that audio scene A and audio scene B share multiple common elements (two audio objects in this example). Since each scene must be complete and independent (for example, for independent reproduction on non-VR playback devices), the common elements must be encoded twice for each scene.

[0221] All audio streams are encoded at different bitrates (i.e. different representations), allowing adaptation based on the effective bitrate of the network connection (i.e. for users on high-speed connections a high bitrate encoded version is delivered, while for users on slow connections a lower bitrate version is delivered).

[0222] • The audio streams are stored on a media server, where for each audio stream different encodings (i.e. different representations) at different bitrates are grouped in one adaptation set with appropriate data signaling the availability of all created adaptation sets.

[0223] In addition, for adaptation sets, the media server receives information about the location "boundaries" of each audio scene and its relationship to each adaptation set (which can contain, for example, complete audio scenes or only individual objects). In this way, each adaptation set can be associated with one of the available audio scenes. The boundaries of an audio scene can be defined, for example, as geometric coordinates of a sphere (e.g., center and radius).

[0224] o Each adaptation set also contains descriptive information about the sound scene or the location where the audio element is active. For example, if an auxiliary stream contains one or several objects, the adaptation set may contain information such as the location where the object is audible (e.g., the coordinates of a sphere center and radius).

[0225] The media server provides information about the location "boundaries" associated with each adaptation set to the client (e.g., a DASH client). For example, in the case of a DASH delivery environment, this can be embedded in the Media Presentation Description (MPD) XML syntax.

[0226] The client receives information about the user's position and / or orientation and / or direction of movement (or any information characterizing a change triggered by the user's actions)

[0227] The client receives information about each adaptation set and, based on this information and the user's position and / or orientation and / or movement direction (or any information characterizing changes triggered by the user's actions, e.g. including x, y, z coordinates and / or yaw, pitch, roll values), the client selects one or more adaptation sets that fully describe the audio scene that should be reproduced at the user's current position.

[0228] The client requests one or more adaptation sets

[0229] In addition, the client can select more adaptation sets to fully describe more than one audio scene and use the audio streams corresponding to more than one audio scene to create a new audio scene that should be reproduced at the user's current location, for example, if the user is walking in the VR environment and at some moment is between two audio scenes (or at a location where two audio scenes have an audible effect).

[0230] o Once the audio streams are available, multiple media decoders may be used to decode the individual audio streams and an additional mixer / renderer 1238 may be used to reproduce the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, certain audio elements that are not audible at that particular position should be disabled or not rendered)

[0231] Alternatively, based on information about the user's position and / or orientation and / or direction of movement, the metadata processor 1236 may be used to manipulate the audio metadata associated with all audio streams to:

[0232] Selecting / enabling the required audio elements 152 for synthesizing a new audio scene;

[0233] And allows all audio streams to be merged into a single audio stream.

[0234] The media server delivers the required adaptation set

[0235] • Alternatively, the client provides information about the user location to the media server, and the media server provides an indication about the required adaptation set.

[0236] Figure 1.2 Another example implementation of such a system is shown, including:

[0237] On the encoding side

[0238] o Multiple media encoders that can be used to create one or more audio streams for each available audio scene associated with a sound scene portion of a viewpoint

[0239] o Multiple media encoders, which can be used to create one or more video streams for each available video scene associated with a video scene portion of a viewpoint. For simplicity, the video encoders are not shown in the figure

[0240] o A media server that stores multiple audio and video adaptation sets, each of which includes different encodings of the same audio and video streams at different bit rates (i.e., different representations). In addition, the media server contains descriptive information for all adaptation sets, which may include

[0241] Availability of all created adaptation sets;

[0242] Information describing the association of an adaptation set with an audio scene and / or viewpoint; in this way, each adaptation set can be associated with one of the available audio scenes;

[0243] Information describing the "bounds" of each audio scene and / or viewpoint (which may contain, for example, a complete audio scene or only individual objects). The bounds of an audio scene may be defined, for example, as geometric coordinates of a sphere (e.g., center and radius).

[0244] The client-side system (client system), which can include any of the following:

[0245] ο The receiving end can receive:

[0246] Information about the user's position and / or orientation and / or direction of movement (or any information characterizing a change triggered by the user's actions)

[0247] Information about the availability of all adaptation sets, and information describing the association of an adaptation set with an audio scene and / or viewpoint; and / or information describing the "boundaries" of each audio scene and / or viewpoint (which may contain, for example, a complete audio scene or only individual objects). For example, in the case of a DASH delivery environment, such information may be provided as part of the Media Presentation Description (MPD) XML syntax.

[0248] o Media consumption device side for content consumption (e.g. based on HMD). The media consumption device is also responsible for collecting information about the user's position and / or orientation and / or movement direction (or any information that characterizes changes triggered by the user's actions)

[0249] o The viewport processor 1232 may be configured to:

[0250] · Receiving information about the current viewport, which may include the user's position and / or orientation and / or movement direction (or any information characterizing changes triggered by the user's actions) from the media consumption device side.

[0251] • Receive information about the ROI signaled in the metadata (like the video viewport signaled in the OMAF specification).

[0252] Receive all available information at the receiving end;

[0253] Decide which audio / video viewpoint should be rendered at a particular moment based on all information received and / or derived from the received and / or available metadata. For example, the viewport processor 1232 may decide:

[0254] A complete audio scene will be reproduced

[0255] A new audio scene must be created from all available audio scenes (e.g., only some audio elements from multiple audio scenes will be reproduced, while the remaining audio elements from these audio scenes will not be reproduced)

[0256] Transitions between two or more audio scenes must be reproduced

[0257] The selection portion 1230 is configured to select one or more adaptation sets signaled in the information received by the receiving end from the available adaptation sets based on the information received from the viewport processor 1232. The selected adaptation set completely describes the audio scene that should be reproduced at the user's current position. The audio scene can be a complete audio scene defined at the encoding side, or a new audio scene that must be created from all available audio scenes.

[0258] ·In addition, in the case when a transition between two or more audio scenes is about to occur based on an indication from the viewport processor 1232, the selection portion can be configured to select one or more adaptation sets signaled in the information received by the receiving end from the available adaptation sets; the selected adaptation set fully describes the audio scenes that may need to be reproduced in the near future (for example, if the user walks in the direction of the next audio scene at a certain speed, it can be predicted that the next audio scene will be necessary and selected before reproduction).

[0259] Furthermore, one may first select some adaptation sets corresponding to neighboring positions at lower bitrates (i.e., select representations encoded at a lower bitrate from among the available representations in an adaptation set), and based on the position change, improve the quality by selecting a higher bitrate for those specific adaptation sets (i.e., select representations encoded at a higher bitrate from among the available representations in an adaptation set).

[0260] o Download and Switch section, can be configured as:

[0261] Requesting one or more of the available adaptation sets from the media server 120 based on the indication received from the selection portion 1230;

[0262] Receiving one or more of the available adaptation sets from the media server 120 (i.e., one representation of all the representations available within each adaptation set);

[0263] Extract metadata information from all received audio streams

[0264] o Metadata processor 1236 may be configured to:

[0265] Receive information about the received audio streams from downloading and switching, which may include audio metadata corresponding to each received audio stream

[0266] Process and manipulate the audio metadata associated with each audio stream based on information received from the viewport processor 1232, which may include information about the user's position and / or orientation and / or direction of movement, to:

[0267] Select / enable the required audio elements 152 for synthesizing the new audio scene indicated by the viewport processor 1232;

[0268] Allows merging all audio streams into a single audio stream.

[0269] o Stream multiplexer / merger 1238 may be configured to merge all selected audio streams into one audio stream based on information received from metadata processor 1236, which may include modified and processed audio metadata corresponding to all received audio streams

[0270] o A media decoder configured to receive and decode at least one audio stream based on information about the user's position and / or orientation and / or movement direction to reproduce a new audio scene indicated by the viewport processor 1232.

[0271] Figure 1.3 A system is shown including a system at the client side (client system), which may be embodied as, for example Figure 1.2 The portion of the system described in , which portion additionally or alternatively includes:

[0272] • Multiple media decoders, which may be configured to decode a single audio stream indicated by the metadata processor 1236 (eg, where multiple audio elements are not activated).

[0273] • A mixer / renderer 1238 which may be configured to reproduce the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, some audio elements that are not audible at that particular position should be disabled or not rendered).

[0274] Option 2

[0275] Figure 1.4 、 Figure 1.5 and Figure 1.6 Refers to an example of solution 2 according to the present invention (which can be Figure 1.1 and / or Figure 1.2 and / or Figure 1.3 Example embodiment of ): independent adaptation sets for discrete positions (viewpoints) with flexible adaptation sets.

[0276] As the user moves in the VR environment, the audio scene 150 may change in a continuous manner. To ensure a good audio experience, all audio elements 152 of the composite audio scene at a particular moment may have to be available to the media decoder, which can utilize the position information to create the final audio scene.

[0277] If the content is pre-encoded for multiple predefined locations, the system can provide accurate reproduction of the audio scenes at these specific locations, provided that these audio scenes do not overlap and the user can "jump / switch" from one location to the next.

[0278] However, in the case where the user "walks" from one location to the next, audio elements from two (or more) audio scenes may be heard simultaneously 152. A solution for this use case was provided in the previous system example, where, regardless of the mechanisms provided for decoding multiple audio streams (using a multiplexer with a single media decoder or multiple media decoders with an attached mixer / renderer 1238), an audio stream describing the complete audio scene 150 must be provided to the client / system 102.

[0279] The following optimization is provided by introducing the concept of common audio elements 152 between multiple audio streams.

[0280] Figure 1.4 An example is shown in which different scenes share at least one audio element (audio object, sound source, etc.). Thus, the client 102 may receive, for example, one main stream 106A associated with only one scene A (e.g., associated with the environment the user is currently in) and associated with object 152A, and one auxiliary stream 106B shared by different scenes B (e.g., a stream in the boundary between scene A the user is currently in and an adjacent or neighboring stream B that shares object 152B) and associated with object 152B.

[0281] Therefore, if Figure 1.4 As shown:

[0282] Multiple independent or overlapping audio scenes are encoded as multiple audio streams. The audio stream 106 is created as follows:

[0283] o For each audio scene 150, a main stream may be created by including only the audio elements 152 that are part of the corresponding audio scene and not part of any other audio scene; and / or

[0284] o For all audio scenes 150 that share an audio element 152, the common audio element 152 may be encoded only in a secondary audio stream associated with only one of the audio scenes, and appropriate metadata information is created indicating the association with the other audio scenes. Or in other words, additional metadata indicates the probability that certain audio streams may be used with multiple audio scenes; and / or

[0285] o Depending on the use case, additional auxiliary streams may be created (e.g., certain audio objects containing different languages ​​may be encoded in separate streams for efficient delivery).

[0286] o In the examples provided:

[0287] Audio scene A is encoded as:

[0288] Main audio stream (A1, 106A),

[0289] Auxiliary audio stream (A2, 106B)

[0290] Metadata information that may indicate that certain audio elements 152B from audio scene A are not encoded in these audio streams A, but are encoded in an auxiliary stream A2 (106B) belonging to a different audio scene (audio scene B)

[0291] Audio scene B is encoded as:

[0292] Main audio stream (B1, 106C),

[0293] Auxiliary audio stream (B2),

[0294] Auxiliary audio stream (B3),

[0295] Metadata information, which may indicate that the audio element 152B from the audio stream B2 is a common audio element 152B that also belongs to the audio scene A.

[0296] • Audio scene C is encoded into three streams (C1, C2 and C3).

[0297] The audio streams 106 (106A, 106B, 106C, ...) may be encoded at different bit rates (i.e., different representations), allowing, for example, adaptation to the effective bit rate of the network connection (i.e., for users with high-speed connections, a high-bit-rate encoded version is delivered, while for users with low-speed network connections, a lower-bit-rate version is delivered).

[0298] The audio streams 106 are stored on the media server 120, where, for each audio stream, different encodings at different bit rates (i.e., different representations) are grouped in an adaptation set with appropriate data signaling the availability of all created adaptation sets. (Multiple representations of a stream associated with the same audio signal but at different bit rates and / or qualities and / or resolutions may be present in the same adaptation set.)

[0299] In addition, for adaptation sets, the media server 120 can receive information about the location "boundaries" of each audio scene and its relationship to each adaptation set (which can contain, for example, complete audio scenes or only individual objects). In this way, each adaptation set can be associated with one or more of the available audio scenes 150. The boundaries of an audio scene can be defined, for example, as geometric coordinates of a sphere (e.g., center and radius).

[0300] o Each adaptation set may also contain descriptive information about the location where the sound scene or audio element 152 is active. For example, if one auxiliary stream (e.g., A2, 106B) contains one or several objects, the adaptation set may contain information such as the location where the object is audible (e.g., coordinates of a sphere center and radius).

[0301] Additionally or alternatively, each adaptation set (e.g., the adaptation set associated with scene B) may contain descriptive information (e.g., metadata) that may indicate that audio elements (e.g., 152B) from one audio scene (e.g., B) are (also or additionally) encoded in an audio stream (e.g., 106B) belonging to a different audio scene (e.g., A).

[0302] The media server 120 may provide information about the location "boundaries" associated with each adaptation set to the system 102 (client), such as a DASH client. For example, in the case of a DASH delivery environment, this may be embedded in the Media Presentation Description (MPD) XML syntax.

[0303] • The system 102 (client) may receive information about the user's position and / or orientation and / or direction of movement (or any information characterizing a change triggered by the user's actions).

[0304] The system 102 (client) may receive information about each adaptation set and, based on this information and the user's position and / or orientation and / or movement direction (or any information characterizing a change triggered by the user's action, e.g., including x, y, z coordinates and / or yaw, pitch, roll values), the system 102 (client) may select one or more adaptation sets that fully or partially describe the audio scene 150 that should be reproduced at the user's 140 current position.

[0305] System 102 (client) may request one or more adaptation sets:

[0306] Additionally, the system 102 (client) may select one or more adaptation sets that fully or partially describe one or more audio scenes 150 and use the audio streams 106 corresponding to the one or more audio scenes 150 to create a new audio scene 150 to be reproduced at the current location of the user 140.

[0307] o Based on metadata indicating that an audio element 152 is part of multiple audio scenes 150, the common audio element 152 may be requested only once to create a new audio scene, rather than requesting the common audio element 152 twice, once for each complete audio scene.

[0308] o Once the audio streams are available to the client system 102, in an example, one or more media decoders (104) may be used to decode the individual audio streams, and / or use additional mixers / renderers to reproduce the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, certain audio elements that are not audible at that particular position should be disabled or not rendered)

[0309] Alternatively or additionally, based on information about the user's position and / or orientation and / or direction of movement, a metadata processor may be used to manipulate the audio metadata associated with all audio streams to:

[0310] Selecting / enabling the required audio elements 152 (152A-152c) for synthesizing a new audio scene; and / or

[0311] And allows all audio streams to be merged into a single audio stream.

[0312] The media server 120 can deliver the required adaptation set

[0313] • Alternatively, the system 102 (client) provides information about the location of the user 140 to the media server 120, and the media server provides an indication of the required adaptation set.

[0314] · Figure 1.5 Another example implementation of such a system is shown, comprising:

[0315] On the encoding side

[0316] o A plurality of media encoders 154 may be used to create one or more audio streams 106 of embedded audio elements 152 based on one or more available audio scenes 150 associated with a sound scene portion of a viewpoint.

[0317] For each audio scene 150, a main stream may be created by including only the audio elements 152 that are part of the corresponding audio scene and not part of any other audio scene.

[0318] Additional auxiliary streams can be created for the same audio scene (e.g., certain audio objects containing different languages ​​can be encoded in separate streams for efficient delivery).

[0319] Additional auxiliary streams may be created, containing:

[0320] Audio elements 152 that are common to more than one audio scene 150

[0321] • Metadata information indicating the association of this auxiliary stream with all other audio scenes 150 that share common audio elements 152. Or in other words, the metadata indicates the probability that certain audio streams can be used with multiple audio scenes.

[0322] o Multiple media encoders, which can be used to create one or more video streams for each available video scene associated with a video scene portion of a viewpoint. For simplicity, the video encoders are not shown in the figure

[0323] o Media server 120 stores multiple audio and video adaptation sets, which include different encodings of the same audio and video streams at different bit rates (i.e., different representations). In addition, media server 120 contains descriptive information for all adaptation sets, which may include

[0324] Availability of all created adaptation sets;

[0325] Information describing the association of an adaptation set with an audio scene and / or viewpoint; in this way, each adaptation set can be associated with one of the available audio scenes;

[0326] Information describing the "bounds" of each audio scene and / or viewpoint (which may contain, for example, a complete audio scene or only individual objects). The bounds of an audio scene may be defined, for example, as geometric coordinates of a sphere (e.g., center and radius).

[0327] • Information indicating the association of one adaptation set with more than one audio scenes, sharing at least one common audio element.

[0328] The client-side system (client system), which can include any of the following:

[0329] ο The receiving end can receive:

[0330] Information about the user's position and / or orientation and / or direction of movement (or any information characterizing a change triggered by the user's actions)

[0331] Information about the availability of all adaptation sets, and information describing the association of an adaptation set with an audio scene and / or viewpoint; and / or information describing the "boundaries" of each audio scene and / or viewpoint (which may contain, for example, a complete audio scene or only individual objects). For example, in the case of a DASH delivery environment, such information may be provided as part of the Media Presentation Description (MPD) XML syntax.

[0332] • Information indicating the association of one adaptation set with more than one audio scenes, sharing at least one common audio element.

[0333] o Media consumption device side for content consumption (e.g. based on HMD). The media consumption device is also responsible for collecting information about the user's position and / or orientation and / or movement direction (or any information that characterizes changes triggered by the user's actions)

[0334] o The viewport processor 1232 may be configured to:

[0335] • Receiving information about the current viewport, which may include the user's position and / or orientation and / or movement direction (or any information characterizing changes triggered by the user's actions) from the media consumption device side.

[0336] • Receive information about the ROI and the ROI signaled in the metadata (such as the video viewport signaled in the OMAF specification).

[0337] Receive all available information at the receiving end;

[0338] Decide which audio / video viewpoint should be rendered at a particular moment based on all information received and / or derived from the received and / or available metadata. For example, the viewport processor 1232 may decide:

[0339] A complete audio scene will be reproduced

[0340] A new audio scene must be created from all available audio scenes (e.g., only some audio elements from multiple audio scenes will be reproduced, while the remaining audio elements from these audio scenes will not be reproduced)

[0341] Transitions between two or more audio scenes must be reproduced

[0342] The selection section 1230 is configured to select one or more adaptation sets from the available adaptation sets based on information received from the viewport processor 1232, signaled from the information received from the receiving end; the selected adaptation set fully or partially describes the audio scene to be reproduced at the user's current position. The audio scene can be a complete or partially complete audio scene defined at the encoding side, or a new audio scene must be created from all available audio scenes.

[0343] Additionally, in case the audio element 152 belongs to more than one audio scene, at least one adaptation set is selected based on information indicating an association of at least one adaptation set with more than one audio scene, the more than one audio scene containing the same audio element 152 .

[0344] ·In addition, in the case when a transition between two or more audio scenes is about to occur based on an indication from the viewport processor 1232, the selection portion can be configured to select one or more adaptation sets signaled in the information received by the receiving end from the available adaptation sets; the selected adaptation set fully describes the audio scenes that may need to be reproduced in the near future (for example, if the user walks in the direction of the next audio scene at a certain speed, it can be predicted that the next audio scene will be necessary and selected before reproduction).

[0345] Furthermore, one may first select some adaptation sets corresponding to neighboring positions at lower bitrates (i.e., select representations encoded at a lower bitrate from among the available representations in an adaptation set), and based on the position change, improve the quality by selecting a higher bitrate for those specific adaptation sets (i.e., select representations encoded at a higher bitrate from among the available representations in an adaptation set).

[0346] o Download and Switch section, can be configured as:

[0347] Requesting one or more of the available adaptation sets from the media server 120 based on the indication received from the selection portion 1230;

[0348] Receiving one or more of the available adaptation sets from the media server 120 (i.e., one representation of all the representations available within each adaptation set);

[0349] Extract metadata information from all received audio streams

[0350] o Metadata processor 1236 may be configured to:

[0351] Receive information about the received audio streams from downloading and switching, which may include audio metadata corresponding to each received audio stream

[0352] Process and manipulate the audio metadata associated with each audio stream based on information received from the viewport processor 1232, which may include information about the user's position and / or orientation and / or direction of movement, to:

[0353] Select / enable the required audio elements 152 for synthesizing the new audio scene indicated by the viewport processor 1232;

[0354] Allows merging all audio streams into a single audio stream.

[0355] o Stream multiplexer / merger 1238 may be configured to merge all selected audio streams into one audio stream based on information received from metadata processor 1236, which may include modified and processed audio metadata corresponding to all received audio streams

[0356] o A media decoder configured to receive and decode at least one audio stream based on information about the user's position and / or orientation and / or movement direction to reproduce a new audio scene indicated by the viewport processor 1232.

[0357] Figure 1.6 A system is shown including a system at the client side (client system), which may be embodied as, for example Figure 5a and Figure 5b The portion of the system described in , which portion additionally or alternatively includes:

[0358] • Multiple media decoders, which may be configured to decode a single audio stream indicated by the metadata processor 1236 (eg, where multiple audio elements are not activated).

[0359] • A mixer / renderer 1238 which may be configured to reproduce the final audio scene based on information about the user's position and / or orientation and / or direction of movement (i.e., for example, some audio elements that are not audible at that particular position should be disabled or not rendered).

[0360] File format updates for file playback

[0361] For the file format use case, multiple main and auxiliary streams can be encapsulated as separate tracks into a single ISOBMFF file. As mentioned before, a single track of such a file will represent a single audio element. Since there is no available MPD that contains the information required for correct playback, this information needs to be provided at the file format level, for example by providing / introducing a specific file format box or multiple specific file format boxes at the track and movie level. Depending on the use case, there is different information required to allow the encapsulated audio scene to be rendered correctly, however the following set of information is fundamental and should therefore always be present:

[0362] Information about the included audio scenes, e.g., "position boundaries"

[0363] Information about all available audio elements, in particular which audio element is packed into which track

[0364] Information about the location of the encapsulated audio element

[0365] A list of all audio elements belonging to an audio scene. An audio element can belong to multiple audio scenes.

[0366] With this information, all mentioned use cases including the one with additional metadata processors and shared encodings should also be possible in a file-based environment.

[0367] Further considerations of the above example

[0368] In the example (e.g., Figures 1.1 to 6 At least one of the scenes may be associated with at least one audio element (audio source 152), each audio element being associated with a location and / or region in the visual environment where the audio element is audible, such that different audio streams are provided from the server system 120 to the client system 102 for different users' locations and / or viewports and / or head orientations and / or movement data and / or interaction metadata and / or virtual position data in the scene.

[0369] In an example, the client system 102 may be configured to decide whether to reproduce at least one audio element 152 and / or an adaptation set of an audio stream (e.g., A1, A2) in the presence of the current user's viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position in the scene, wherein the system 102 is configured to request and / or receive at least one audio element at the current user's virtual position.

[0370] In an example, a client system (e.g., 102) may be configured to predictively determine whether at least one audio element (152) of an audio stream and / or an adaptation set will become relevant and / or audible based at least on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110), and wherein the system is configured to request and / or receive the at least one audio element and / or audio stream and / or adaptation set at a particular user's virtual position prior to the predicted movement and / or interaction of the user in the scene, and wherein the system is configured to reproduce the at least one audio element and / or audio stream at the particular user's virtual position upon receipt of the at least one audio element and / or voice stream after the user's movement and / or interaction in the scene. See, e.g., above Figure 8a and Figure 8b In some examples, at least one of the operations of system 102 or 120 can be performed based on predictive and / or statistical and / or aggregated data.

[0371] In an example, a client system (e.g., 102) may be configured to request and / or receive at least one audio element (e.g., 152) at a lower bit rate and / or quality level at a user's virtual location prior to the user's movement and / or interaction in the scene, wherein the system is configured to request and / or receive at least one audio element at a higher bit rate and / or quality level at the user's virtual location after the user's movement and / or interaction in the scene. See, e.g., Figure 7b .

[0372] In an example, at least one audio element may be associated with at least one scene, at least one audio element associated with a location and / or region in a visual environment associated with the scene, wherein the system is configured to request different streams at different bit rates and / or quality levels for the audio elements based on the relevance and / or audibility level of the audio elements at each user's virtual location in the scene, wherein the system is configured to request audio streams at higher bit rates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location, and / or request audio streams at lower bit rates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location. Generally speaking, see Figure 7a See also Figure 2a and Figure 2b (wherein the more relevant and / or audible source may be closer to the user), Figure 3 (wherein the more relevant and / or audible source is the source of scene 150a when the user is in position x1, and the more relevant and / or audible source is the source of scene 150b when the user is in position x3), Figure 4 (wherein, at time t2, the more relevant and / or audible source may be the source of the first scene), Figure 6 (Where the larger audible source may be the source the user is looking directly at).

[0373] In an example, at least one audio element (152) is associated with a scene, each audio element being associated with a location and / or region in a visual environment associated with the scene, wherein the client system 102 is configured to periodically send the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) to the server system 120 to: provide a higher bitrate and / or quality stream from the server for locations close to the at least one audio element (152) and provide a lower bitrate and / or quality stream from the server for locations further from the at least one audio element (152). See, e.g. Figure 2a and Figure 2b .

[0374] In an example, multiple scenes (e.g., 150A, 150B) may be defined for multiple visual environments such as adjacent and / or neighboring environments, thereby providing a first stream associated with a first current scene (e.g., 150A), and in the event of a user transition (150AB) to another second scene (e.g., 150B), providing both a stream associated with the first scene and a second stream associated with the second scene. See, e.g., Figure 3 .

[0375] In an example, a plurality of scenes are defined for a first and a second visual environment, the first and the second environments being adjacent and / or neighboring environments, wherein, in a case where the user's virtual position is in a first environment associated with the first scene, a first stream associated with the first scene is provided from a server for reproducing the first scene, in a case where the user's virtual position is in a second environment associated with the second scene, a second stream associated with the second scene is provided from the server for reproducing the second scene, and in a case where the user's virtual position is in a transition position between the first scene and the second scene, both the first stream associated with the first scene and the second stream associated with the second scene are provided. See, e.g. Figure 3 .

[0376] In an example, when the user is in a first environment associated with a first scene, a first stream associated with the first scene is obtained at a higher bit rate and / or quality, and when the user is at the beginning of a transition position from the first scene to the second scene, a second stream associated with a second environment is obtained at a lower bit rate and / or quality, and when the user is at the end of a transition position from the first scene to the second scene, the first stream associated with the first scene is obtained at a lower bit rate and / or quality, and the second stream associated with the second scene is obtained at a higher bit rate and / or quality. This can be, for example Figure 3 situation.

[0377] In an example, multiple scenes (e.g., 150A, 150B) are defined for multiple visual environments (e.g., the neighboring environment) such that system 102 can request and / or obtain a stream associated with the current scene at a higher bitrate and / or quality, and request and / or obtain a stream associated with a second scene at a lower bitrate and / or quality. See, for example Figure 4 .

[0378] In an example, multiple N audio elements are defined, and in a case where the distance of a user to the location or area of these audio elements is greater than a predetermined threshold, the N audio elements are processed to obtain a smaller number M of audio elements (M < N) associated with the location or area close to the location or area of the N audio elements, to provide at least one audio stream associated with the N audio elements to the system in a case where the distance of the user to the location or area of the N audio elements is less than the predetermined threshold, or to provide at least one audio stream associated with the M audio elements to the system in a case where the distance of the user to the location or area of the N audio elements is greater than the predetermined threshold. See, for example Figure 1.7 .

[0379] In an example, at least one visual environment scene is associated with at least one of multiple N audio elements (N >= 2), each audio element being associated with a location and / or area in the visual environment, wherein at least one of the multiple N audio elements can be provided in at least one representation at a high bitrate and / or quality level, and wherein at least one of the multiple N audio elements is provided in at least one representation at a low bitrate and / or quality level, wherein at least one representation is obtained by processing the N audio elements to obtain a smaller number M of audio elements (M < N) associated with the location or area close to the location or area of the N audio elements, wherein the system is configured to request a representation at a higher bitrate and / or quality level for an audio element in a case where the audio element is more relevant and / or more audible at the virtual location of the current user in the scene, and wherein the system is configured to request a representation at a lower bitrate and / or quality level for an audio element in a case where the audio element is less relevant and / or less audible at the virtual location of the current user in the scene. See, for example Figure 1.7 .

[0380] In an example, different streams are obtained for different audio elements in a case where the distance and / or relevance and / or audibility level and / or angular orientation of the user is below a predetermined threshold. See, for example Figure 1.7 .

[0381] In an example, different audio elements are provided at different viewports such that in a case where a first audio element falls within the current viewport, the first audio element is obtained at a higher bitrate than a second audio element that does not fall within the viewport. See, for example Figure 6 .

[0382] In an example, at least two visual environment scenes are defined, wherein at least one first and second audio elements are associated with a first scene associated with the first visual environment, and at least one third audio element is associated with a second scene associated with the second visual environment, wherein the system 102 is configured to obtain metadata describing that the at least one second audio element is additionally associated with the second visual environment scene, and wherein, in a case where the user's virtual position is in the first visual environment, the system is configured to request and / or receive at least the first and second audio elements, and wherein, in a case where the user's virtual position is in the second visual environment scene, the system is configured to request and / or receive at least the second and third audio elements, and wherein, in a case where the user's virtual position is in transition between the first visual environment scene and the second visual environment scene, the system is configured to request and / or receive at least the first, second, and third audio elements. See, e.g. Figure 1.4 This can also be applied to Figure 3 .

[0383] In an example, at least one first audio element may be provided in at least one audio stream and / or adaptation set, and at least one second audio element may be provided in at least one second audio stream and / or adaptation set, and at least one third audio element may be provided in at least one third audio stream and / or adaptation set, and wherein at least the first visual environment scene is described by metadata as a complete scene requiring at least the first and second audio streams and / or adaptation sets, and wherein the second visual environment scene is described by metadata as an incomplete scene requiring at least the third audio stream and / or adaptation set and at least the second audio stream and / or adaptation set associated with at least the first visual environment scene, wherein the system includes a metadata processor configured to manipulate the metadata to allow merging the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment into a new single stream if the user's virtual position is in the second visual environment. See, e.g. Figure 1.2 、 Figure 1.3 、 Figure 1.5 and Figure 1.6 .

[0384] In an example, the system 102 may include a metadata processor (e.g., 1236) configured to manipulate metadata in at least one audio stream prior to at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0385] In an example, a metadata processor (e.g., 1236) may be configured to enable and / or disable at least one audio element in at least one audio stream before at least one audio decoder based on a user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, wherein the metadata processor may be configured to disable at least one audio element in at least one audio stream before at least one audio decoder if the system decides not to reproduce the audio element due to the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and wherein the metadata processor may be configured to enable at least one audio element in at least one audio stream before at least one audio decoder if the system decides to reproduce the audio element due to the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0386] Server side

[0387] The above also relates to a server (120) for delivering audio and video streams to a client for a virtual reality VR, augmented reality AR, mixed reality MR or 360-degree video environment, the video and audio streams to be reproduced in a media consumption device, wherein the server (120) comprises: an encoder for encoding and / or a memory for storing a video stream describing a visual environment, the visual environment being associated with an audio scene; wherein the server further comprises an encoder for encoding and / or a memory for storing a plurality of streams and / or audio elements and / or adaptation sets to be delivered to the client, the streams and / or audio elements and / or adaptation sets being associated with at least one audio scene, wherein the server is configured to:

[0388] selecting and delivering a video stream based on a request from a client, the video stream being associated with the environment;

[0389] selecting an audio stream and / or an audio element and / or an adaptation set based on a request from the client, the request being associated with at least a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data of the user and with an audio scene associated to the environment; and

[0390] The audio stream is delivered to the client.

[0391] Further embodiments and modifications

[0392] Depending on the specific implementation requirements, examples may be implemented in hardware. Implementations may be performed using a digital storage medium (e.g., a floppy disk, a digital versatile disk (DVD), a Blu-ray disk, a compact disk (CD), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or flash memory) having stored thereon electronically readable control signals that cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer-readable.

[0393] Generally, the examples can be implemented as a computer program product having program instructions, the program instructions being operable to perform one of the methods when the computer program product runs on a computer. The program instructions can, for example, be stored on a machine-readable medium.

[0394] Other examples include a computer program stored on a machine-readable carrier for performing one of the methods described herein. In other words, a method example is therefore a computer program having program instructions for performing one of the methods described herein when the computer program runs on a computer.

[0395] Therefore, another example of a method is a data carrier medium (or digital storage medium or computer-readable medium) on which is recorded a computer program for performing one of the methods described herein. The data carrier medium, digital storage medium or recorded medium is tangible and / or non-transferable, rather than an intangible and transitory signal.

[0396] Another example comprises a processing unit, for example a computer or a programmable logic device, which performs one of the methods described herein.

[0397] A further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0398] Another example includes an apparatus or system for transmitting a computer program to a receiver (e.g., electronically or optically), the computer program for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0399] In some examples, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some examples, a field programmable gate array can cooperate with a microprocessor to perform the methods described herein. Generally, these methods are performed by any suitable hardware device.

[0400] The above examples are illustrative of the principles disclosed above. It should be understood that modifications and variations of the arrangements and details described herein will be apparent. Accordingly, it is intended that the scope of the appended patent claims be limited rather than by the specific details provided by way of description and explanation of the examples herein.

[0401] Example embodiment 1, a system (102) for virtual reality VR, augmented reality AR, mixed reality MR or 360-degree video environment, configured to receive video and audio streams to be reproduced in a media consumption device,

[0402] wherein the system (102) comprises:

[0403] at least one media video decoder configured to decode a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to a user, and

[0404] at least one audio decoder (104) configured to decode an audio signal (108) from at least one audio stream (106),

[0405] The system (102) is configured to request (112) at least one audio stream (106) and / or an audio element of an audio stream and / or an adaptation set from a server (120) based at least on a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) of a user.

[0406] Example embodiment 2, according to the system of example embodiment 1, is configured to: provide the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110) to the server (120) to obtain at least one audio stream (106) and / or an audio element of an audio stream and / or an adaptation set from the server (120).

[0407] Example embodiment 3. A system according to example embodiment 1 or 2, wherein at least one scene is associated with at least one audio element (152), each audio element being associated with a position and / or area in a visual environment in which the audio element is audible, so that different audio streams are provided for different positions and / or viewports and / or head orientations and / or movement data and / or interaction metadata and / or virtual position data of different users in the scene.

[0408] Example embodiment 4, the system according to any of the preceding example embodiments, configured to: determine whether to reproduce at least one audio element and / or an adaptation set of an audio stream for the viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position of the current user in the scene, and

[0409] The system is configured to request and / or receive the at least one audio element at the virtual position of the current user.

[0410] Example embodiment 5. A system according to any of the preceding example embodiments, wherein the system is configured to: predictively determine whether at least one audio element (152) of an audio stream and / or an adaptation set will become relevant and / or audible based on at least the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data (110), and

[0411] wherein the system is configured to request and / or receive at least one audio element and / or audio stream and / or adaptation set at a virtual position of a specific user prior to a predicted movement and / or interaction of the user in the scene,

[0412] The system is configured to reproduce the at least one audio element and / or audio stream at the virtual position of the specific user when the at least one audio element and / or audio stream is received after the user moves and / or interacts in the scene.

[0413] Example embodiment 6. The system of any of the preceding example embodiments, configured to: request and / or receive the at least one audio element (152) at a lower bit rate and / or quality level at the user's virtual location prior to the user's movement and / or interaction in the scene,

[0414] The system is configured to request and / or receive the at least one audio element at a higher bit rate and / or quality level at the user's virtual location following the user's movement and / or interaction in the scene.

[0415] Example embodiment 7. The system of any of the preceding example embodiments, wherein at least one audio element (152) is associated with at least one scene, each audio element being associated with a location and / or region in the visual environment associated with the scene,

[0416] The system is configured to request and / or receive streams at a higher bit rate and / or quality for audio elements closer to the user than for audio elements further away from the user.

[0417] Example embodiment 8. The system of any one of the preceding example embodiments, wherein at least one audio element (152) is associated with at least one scene, the at least one audio element being associated with a location and / or area in the visual environment associated with the scene,

[0418] wherein the system is configured to request different streams at different bit rates and / or quality levels for audio elements based on the relevance and / or audibility level of the audio elements at the virtual position of each user in the scene,

[0419] wherein the system is configured to: request audio streams at higher bit rates and / or quality levels for audio elements that are more relevant and / or more audible at the current user's virtual location, and / or

[0420] Audio streams are requested at lower bit rates and / or quality levels for audio elements that are less relevant and / or less audible at the current user's virtual location.

[0421] Example embodiment 9. The system of any of the preceding example embodiments, wherein at least one audio element (152) is associated with a scene, each audio element being associated with a location and / or region in the visual environment associated with the scene,

[0422] The system is configured to periodically send the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual location data (110) to the server, such that:

[0423] providing a higher bitrate and / or quality stream from the server for the first location, and

[0424] providing a lower bitrate and / or quality stream from the server for a second location,

[0425] Wherein the first position is closer to the at least one audio element (152) than the second position.

[0426] Example embodiment 10. The system of any one of the preceding example embodiments, wherein a plurality of scenes (150A, 150B) are defined for a plurality of visual environments, such as adjacent and / or neighboring environments,

[0427] Such that a first stream associated with a first current scene is provided, and in case of a user transition to a second other scene, both a stream associated with said first scene and a second stream associated with said second scene are provided.

[0428] Example embodiment 11. The system of any of the preceding example embodiments, wherein a plurality of scenes (150A, 150B) are defined for first and second visual environments, the first and second environments being adjacent and / or neighboring environments,

[0429] wherein, when the user's location or virtual location is in a first environment associated with the first scene, a first stream associated with the first scene is provided from the server to reproduce the first scene;

[0430] providing a second stream associated with the second scene from the server to reproduce the second scene in a case where the user's position or virtual position is in a second environment associated with the second scene, and

[0431] In case the position or virtual position of the user is in a transition position between the first scene and the second scene, both a first stream associated with the first scene and a second stream associated with the second scene are provided.

[0432] Example embodiment 12. The system of any one of the preceding example embodiments, wherein a plurality of scenes (150A, 150B) are defined for first and second visual environments, the first and second visual environments being adjacent and / or neighboring environments,

[0433] The system is configured to: when the user's virtual location is in the first environment, request and / or receive a first stream associated with a first scene (150A) associated with the first environment for reproducing the first scene,

[0434] The system is configured to: request and / or receive a second stream associated with a second scene (150B) associated with the second environment in order to reproduce the second scene, when the user's virtual location is in the second environment; and

[0435] The system is configured to request and / or receive both a first stream associated with the first scene and a second stream associated with the second scene when the user's virtual position is in a transition position (150AB) between the first environment and the second environment.

[0436] Example embodiment 13. The system of any one of example embodiments 10 to 12, wherein:

[0437] When the user is in the first environment associated with the first scene, obtaining the first stream associated with the first scene at a higher bit rate and / or quality,

[0438] obtaining the second stream associated with the second scene associated to the second environment at a lower bit rate and / or quality when the user is at the beginning of a transition position from the first scene to the second scene, and

[0439] When the user is at the end of the transition position from the first scene to the second scene, obtain the first stream associated with the first scene at a lower bitrate and / or quality, and obtain the second stream associated with the second scene at a higher bitrate and / or quality,

[0440] where the lower bitrate and / or quality is lower than the higher bitrate and / or quality.

[0441] Example embodiment 14. The system according to any of the foregoing example embodiments, wherein a plurality of scenes (150A, 150B) are defined for a plurality of environments, which may be, for example, adjacent and / or neighboring environments,

[0442] such that the system is configured to obtain a stream associated with a first current scene associated with a first current environment, and

[0443] in a case where the distance of the user's position or virtual position from the boundary of the scene is lower than a predetermined threshold, the system further obtains an audio stream associated with a second neighboring and / or adjacent environment associated with the second scene.

[0444] Example embodiment 15. The system according to any of the foregoing example embodiments, wherein a plurality of scenes (150A, 150B) are defined for a plurality of visual environments,

[0445] such that the system requests and / or obtains a stream associated with the current scene at a higher bitrate and / or quality, and requests and / or obtains a stream associated with the second scene at a lower bitrate and / or quality,

[0446] where the lower bitrate and / or quality is lower than the higher bitrate and / or quality.

[0447] Example embodiment 16. The system according to any of the foregoing example embodiments, wherein a plurality of N audio elements are defined, and in a case where the distance of the user from the position or area of these audio elements is greater than a predetermined threshold, process the N audio elements to obtain a smaller number M of audio elements (M < N) associated with a position or area close to the position or area of the N audio elements, in order to:

[0448] in a case where the distance of the user from the position or area of the N audio elements is less than a predetermined threshold, provide at least one audio stream associated with the N audio elements to the system, or

[0449] in a case where the distance of the user from the position or area of the N audio elements is greater than a predetermined threshold, provide at least one audio stream associated with the M audio elements to the system.

[0450] Example Embodiment 17, a system according to any of the foregoing example embodiments, wherein at least one visual environment scene is associated with at least one of a plurality of N audio elements (N >= 2), each audio element being associated with a position and / or region in the visual environment,

[0451] wherein at least the at least one of the plurality of N audio elements is provided in at least one representation at a high bitrate and / or quality level, and

[0452] wherein at least the at least one of the plurality of N audio elements is provided in at least one representation at a low bitrate and / or quality level, wherein at least one representation is obtained by processing the N audio elements to obtain a smaller number M of audio elements (M < N) associated with a position or region proximate to the position or region of the N audio elements,

[0453] wherein the system is configured to request the representation at a higher bitrate and / or quality level for the audio element when the audio element is more relevant and / or more audible at the virtual position of the current user in the scene,

[0454] wherein the system is configured to request the representation at a lower bitrate and / or quality level for the audio element when the audio element is less relevant and / or less audible at the virtual position of the current user in the scene.

[0455] Example Embodiment 18, a system according to Example Embodiments 16 and 17, wherein different streams are obtained for different audio elements when the distance and / or relevance and / or audibility level and / or angular orientation of the user is below a predetermined threshold.

[0456] Example Embodiment 19, a system according to any of the foregoing example embodiments, wherein the system is configured to request and / or obtain the stream based on the orientation of the user in the scene and / or the direction of movement of the user and / or user interaction.

[0457] Example Embodiment 20, a system according to any of the foregoing example embodiments, wherein the viewport is associated with the position and / or virtual position and / or movement data and / or head.

[0458] Example embodiment 21, a system according to any of the preceding example embodiments, wherein different audio elements are provided at different viewports, wherein the system is configured to: in a case where a first audio element (S1) falls within the viewport (160-1), request and / or receive the first audio element at a higher bit rate than a second audio element (S2) that does not fall within the viewport.

[0459] Example embodiment 22, the system of any one of the preceding example embodiments, configured to request and / or receive a first audio stream and a second audio stream, wherein the first audio element in the first audio stream is more relevant and / or more audible than the second audio element in the second audio stream,

[0460] The first audio stream is requested and / or received at a higher bit rate and / or quality than the second audio stream.

[0461] Example embodiment 23. The system of any one of the preceding example embodiments, wherein at least two visual environment scenes are defined, wherein at least one first and second audio element are associated with a first scene associated with the first visual environment, and at least one third audio element is associated with a second scene associated with the second visual environment,

[0462] wherein the system is configured to obtain metadata describing that the at least one second audio element is additionally associated with the second visual environment scene, and

[0463] wherein the system is configured to request and / or receive the at least first and second audio elements if the user's virtual position is within the first visual environment, and

[0464] wherein the system is configured to request and / or receive the at least second and third audio elements if the user's virtual position is within the second visual environment scene, and

[0465] The system is configured to request and / or receive the at least first, second and third audio elements when the user's virtual position is in transition between the first visual environment scene and the second visual environment scene.

[0466] Example embodiment 24. The system of example embodiment 23, wherein the at least one first audio element is provided in at least one audio stream and / or adaptation set, and the at least one second audio element is provided in at least one second audio stream and / or adaptation set, and the at least one third audio element is provided in at least one third audio stream and / or adaptation set, and wherein the at least first visual environment scene is described by metadata as a complete scene requiring the at least first and second audio streams and / or adaptation sets, and wherein the second visual environment scene is described by metadata as an incomplete scene requiring the at least third audio stream and / or adaptation set and the at least second audio stream and / or adaptation set associated with the at least first visual environment scene.

[0467] The system includes a metadata processor configured to manipulate metadata to allow the second audio stream belonging to the first visual environment and the third audio stream associated with the second visual environment to be merged into a new single stream when the user's virtual position is in the second visual environment.

[0468] Example embodiment 25, a system according to any of the preceding example embodiments, wherein the system includes a metadata processor configured to manipulate metadata in at least one audio stream before the at least one audio decoder based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0469] Example embodiment 26. The system of example embodiment 25, wherein the metadata processor is configured to: enable and / or disable at least one audio element in at least one audio stream before the at least one audio decoder based on the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, wherein

[0470] The metadata processor is configured to disable at least one audio element in at least one audio stream before the at least one audio decoder if the system decides not to reproduce the audio element due to the current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data, and wherein

[0471] The metadata processor is configured to enable at least one audio element in at least one audio stream before the at least one audio decoder if the system determines that the audio element is to be reproduced due to the user's current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data.

[0472] Example embodiment 27, the system according to any of the preceding example embodiments, is configured to disable decoding of audio elements selected based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position.

[0473] Example Embodiment 28. The system according to any one of the preceding example embodiments, configured to merge at least one first audio stream associated with the current audio scene into at least one stream associated with an adjacent, neighboring and / or future audio scene.

[0474] Example embodiment 29, according to any one of the preceding example embodiments, the system is configured to obtain and / or collect statistical or aggregated data about the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual location data to send a request associated with the statistical or aggregated data to the server.

[0475] Example embodiment 30, a system according to any of the preceding example embodiments, is configured to deactivate decoding and / or reproduction of the at least one stream based on metadata associated with the at least one stream and based on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data.

[0476] Example embodiment 31, the system according to any one of the preceding example embodiments, is further configured to:

[0477] manipulating metadata associated with the selected group of audio streams based at least on the user's current or estimated viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to:

[0478] Selecting and / or enabling and / or activating audio elements that synthesize said audio scene to be reproduced; and / or

[0479] Enables merging of all selected audio streams into a single audio stream.

[0480] Example embodiment 32, a system according to any of the preceding example embodiments, is configured to control the request for the at least one stream to the server based on the distance of the user's location from the boundaries of adjacent and / or neighboring environments associated with different scenes or other metrics associated with the user's location in the current environment or predictions of future environments.

[0481] Example embodiment 33, a system according to any of the preceding example embodiments, wherein for each audio element or audio object, information is provided from the server system (120), wherein the information includes descriptive information about the location where the sound scene or the audio element is active.

[0482] Example embodiment 34, a system according to any of the preceding example embodiments, configured to select between reproducing one scene and synthesizing or mixing or multiplexing or superimposing or combining at least two scenes based on current or future or viewport and / or head orientation and / or movement data and / or metadata and / or virtual position and / or user's selection, the two scenes being associated with different adjacent and / or neighboring environments.

[0483] Example embodiment 35, the system according to any one of the preceding example embodiments, configured to at least create or use the adaptation set such that:

[0484] Multiple adaptation sets are associated with one audio scene; and / or

[0485] Additional information is provided associating each adaptation set with a viewpoint or an audio scene; and / or

[0486] Provide additional information that can include the following:

[0487] - information about the boundaries of an audio scene and / or

[0488] - information about the relationship between an adaptation set and an audio scene (e.g. an audio scene is encoded in three streams which are encapsulated in three adaptation sets) and / or

[0489] - Information about connections between boundaries of the audio scene and the plurality of adaptation sets.

[0490] Example embodiment 36, the system according to any one of the preceding example embodiments, configured to:

[0491] receiving a stream of scenes associated with an adjacent or proximate environment;

[0492] Upon detecting a transition of a boundary between two environments, decoding and / or rendering of the stream for the adjacent or neighboring environment is started.

[0493] Example embodiment 37, a system comprising the system of any of the preceding example embodiments, the system being configured to operate as a client and a server, the server being configured to deliver video and / or audio streams to be reproduced in a media consumption device.

[0494] Example embodiment 38. The system of any one of the preceding example embodiments, wherein the system is further configured to:

[0495] requesting and / or receiving at least one first adaptation set, the first adaptation set comprising at least one audio stream associated with at least one first audio scene;

[0496] Requesting and / or receiving at least one second adaptation set, the second adaptation set comprising at least one second audio stream associated with at least two audio scenes, the at least two audio scenes comprising at least one first audio scene; and

[0497] The at least one first audio stream is enabled to be merged with the at least one second audio stream into a new audio stream to be decoded based on available metadata related to the following items: the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or information describing the association of the at least one first adaptation set with the at least one first audio scene and / or the association of the at least one second adaptation set with the at least one first audio scene.

[0498] Example embodiment 39, the system according to any one of the preceding example embodiments, configured to:

[0499] receiving information about a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or any information characterizing changes triggered by actions of said user; and

[0500] Information about the availability of adaptation sets and information describing an association of at least one adaptation set with at least one scene and / or viewpoint and / or viewport and / or position and / or virtual position and / or movement data and / or orientation is received.

[0501] Example embodiment 40, the system according to any one of the preceding example embodiments, configured to:

[0502] deciding whether to reproduce at least one audio element from at least one audio scene embedded in at least one stream and at least one additional audio element from at least one additional audio scene embedded in at least one additional stream; and

[0503] In case of a positive decision, an operation is caused of merging or compositing or multiplexing or superimposing or combining the at least one additional stream of the additional audio scene to the at least one stream of the at least one audio scene.

[0504] Example embodiment 41, the system according to any one of the preceding example embodiments, configured to:

[0505] manipulating audio metadata associated with a selected audio stream based at least on the user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to:

[0506] selecting and / or enabling and / or activating said audio elements of said audio scene that are synthesized to be reproduced; and

[0507] Enables merging of all selected audio streams into a single audio stream.

[0508] Example embodiment 42, a server (120) for delivering audio and video streams to a client for virtual reality VR, augmented reality AR, mixed reality MR or 360-degree video environment, wherein the video and audio streams are to be reproduced in a media consumption device,

[0509] wherein the server (120) comprises an encoder for encoding and / or a memory for storing a video stream describing a visual environment, the visual environment being associated with an audio scene;

[0510] The server further comprises an encoder for encoding and / or a memory, wherein the memory is used to store multiple streams and / or audio elements and / or adaptation sets to be delivered to the client, wherein the streams and / or audio elements and / or adaptation sets are associated with at least one audio scene.

[0511] Wherein, the server is configured as:

[0512] selecting and delivering a video stream based on a request from a client, the video stream being associated with the environment;

[0513] selecting an audio stream and / or an audio element and / or an adaptation set based on a request from the client, the request being associated with at least a current viewport and / or head orientation and / or movement data and / or interaction metadata and / or virtual position data of the user and with an audio scene associated to the environment; and

[0514] The audio stream is delivered to the client.

[0515] Example embodiment 43. The server of example embodiment 42, wherein the streams are encapsulated into adaptation sets, each adaptation set comprising multiple streams associated with different representations of the same audio content having different bit rates and / or qualities,

[0516] The selected adaptation set is selected based on a request from the client.

[0517] Example embodiment 44, a system comprising the system of any one of example embodiments 1-41 operating as a client and a server.

[0518] Example embodiment 45, the system according to example embodiment 44, comprising the server according to example embodiment 42 or 43.

[0519] Example 46, a method for virtual reality (VR), augmented reality (AR), mixed reality (MR), or a 360-degree video environment, the method being configured to receive a video and / or audio stream to be reproduced in a media consumption device, comprising:

[0520] Decoding a video signal from a video stream to represent a VR, AR, MR, or 360-degree video environment scene to the user, and

[0521] Decode the audio signal from the audio stream,

[0522] At least one audio stream is requested from and / or obtained from the server based on the user's current viewport and / or position data and / or head orientation and / or movement data and / or metadata and / or virtual position data and / or metadata.

[0523] Example 47, a computer program comprising instructions, which, when executed by a processor, cause the processor to perform the method of Example 46.

Claims

1. A system for receiving an audio stream to be reproduced in a media consumption device, The system comprises: at least one audio decoder configured to decode an audio signal from an audio stream to represent an audio scene, The system is configured to: request and / or receive a first audio stream as a first representation of a first adaptation set and a second audio stream as a second representation of a second adaptation set based at least on movement data and / or location data of a user, wherein the first representation and the second representation are audio streams of different bit rates in the first adaptation set and the second adaptation set, respectively. wherein a first audio element in the first audio stream is more relevant and / or more audible than a second audio element in the second audio stream, wherein the first audio stream is the first representation having a higher bit rate than the second audio stream being the second representation, wherein at least one first audio element of the first audio elements is associated only with a first audio scene, and at least one second audio element of the second audio elements is associated only with the first audio scene, and at least one third audio element is associated only with a second audio scene, wherein the system is configured to obtain interaction metadata describing that the at least one second audio element is additionally associated with the second audio scene, The system is configured to request and / or receive the at least one first audio element and the at least one second audio element while reproducing the first audio scene. wherein the system is configured to request and / or receive the at least one second audio element and the at least one third audio element in case of reproducing the second audio scene, and Therein, the system is configured to request and / or receive the at least one first audio element, the at least one second audio element and the at least one third audio element in case of a transition between the first audio scene and the second audio scene. 2 . The system according to claim 1 , further configured to output current movement data and / or location data of a user in order to obtain the first audio stream and the second audio stream.

3. The system according to claim 1 , further comprising: a method for determining whether at least one audio element of the first adaptation set or the second adaptation set is to be reproduced for current movement data and / or position data of a user in an audio scene; The system is configured to request and / or receive at least one audio element at the user's current movement data and / or location data.

4. The system of claim 1 , further comprising: a system for predictively deciding, based at least on current movement data and / or location data of a user, whether at least one audio element of the first adaptation set or the second adaptation set will become relevant and / or audible. wherein the system is configured to request and / or receive the at least one audio element prior to a predicted movement and / or interaction of the user in the audio scene, and The system is configured to reproduce the at least one audio element upon receipt thereof following movement and / or interaction of the user in the audio scene.

5. The system according to claim 1 , further comprising: requesting and / or receiving audio elements at a lower bit rate at a user's virtual position prior to a user's interaction resulting from a change in position data within the same audio scene or entering a next scene separate from the current audio scene; in, The system is configured to request and / or receive the audio element at a higher bit rate following an interaction of the user in the audio scene.

6. The system according to claim 1, wherein at least one audio element is associated with at least one audio scene, the at least one audio element being associated with specific position data and / or movement data, in, The system is configured to request, for an audio element, different streams as different representations of the first adaptation set at different bit rates based on the relevance and / or audibility level of the audio element at the virtual position of each user in the audio scene, wherein the system is configured to request the first audio stream as the first representation of the first adaptation set at a higher bit rate for audio elements that are more relevant and / or more audible at the user's current virtual position, and / or For audio elements that are less relevant and / or less audible at the user's current virtual position, the second audio stream as the second representation of the second adaptation set is requested at a lower bit rate.

7. The system according to claim 1, wherein at least one audio element is associated with the audio scene, each audio element being associated with specific position data and / or movement data, in, The system is configured to periodically output the user's current movement data and / or location data such that: providing the first audio stream as the first representation of the first adaptation set at a higher bit rate for a first position, and providing, for a second position, the second audio stream as the second representation of the second adaptation set at a lower bit rate, The first position is closer to the audio element than the second position.

8. The system according to claim 1, wherein: defining a plurality of audio scenes for a plurality of audio scenes, the plurality of audio scenes being adjacent and / or neighboring audio scenes, The first audio stream associated with the current first audio scene is provided, and in case of a user transition to another second audio scene, both the first audio stream associated with the first audio scene and the second audio stream associated with the second audio scene are provided.

9. The system according to claim 1, wherein: defining a plurality of audio scenes for a first audio scene and a second audio scene, the first audio scene and the second audio scene being adjacent and / or neighboring audio scenes, wherein, when the position or virtual position of the user is in the first audio scene, providing the first audio stream associated with the first audio scene to reproduce the first audio scene; providing the second audio stream associated with the second audio scene to reproduce the second audio scene when the user's position or virtual position is in the second audio scene, and In case the position or virtual position of the user is in a transition position between the first audio scene and the second audio scene, both the first audio stream associated with the first audio scene and the second audio stream associated with the second audio scene are provided.

10. The system according to claim 1, wherein: defining a plurality of audio scenes for a first audio scene and a second audio scene, the first audio scene and the second audio scene being adjacent and / or neighboring audio scenes, wherein the system is configured to request and / or receive the first audio stream associated with the first audio scene, in case the virtual position of the user is in the first audio scene, to reproduce the first audio scene, The system is configured to request and / or receive the second audio stream associated with the second audio scene to reproduce the second audio scene when the virtual position of the user is in the second audio scene, and wherein the system is configured to request and / or receive both the first audio stream associated with the first audio scene and the second audio stream associated with the second audio scene if the user's virtual position is in a transition position between the first audio scene and the second audio scene.

11. The system according to claim 8, wherein When the user is in the first audio scene, obtaining the first audio stream associated with the first audio scene at a higher bit rate, When the user is at the beginning of the transition from the first audio scene to the second audio scene, the second audio stream associated with the second audio scene is obtained at a lower bit rate, When the user is at an end of a transition position from the first audio scene to the second audio scene, the first audio stream associated with the first audio scene is obtained at a lower bit rate, and the second audio stream associated with the second audio scene is obtained at a higher bit rate, and The lower bit rate is lower than the higher bit rate.

12. The system according to claim 1, wherein: Define multiple audio scenes, The system is configured to obtain an audio stream associated with a current first audio scene, and In case the distance of the user's position or virtual position from the boundary of the audio scene is below a predetermined threshold, the system further obtains an audio stream associated with a neighboring and / or adjacent second audio scene.

13. The system of claim 1, wherein: Define multiple audio scenes, causing the system to request and / or obtain the first audio stream associated with the current audio scene as the first representation of the first adaptation set at a higher bit rate, and to request and / or obtain the second audio stream associated with the second audio scene as the second representation of the second adaptation set at a lower bit rate, and The lower bit rate is lower than the higher bit rate.

14. The system of claim 1, configured to request and / or receive the first audio stream of the first adaptation set and the second audio stream of the second adaptation set, wherein: the first audio element in the first audio stream of the first adaptation set is more relevant and / or more audible than the second audio element in the second audio stream of the second adaptation set, The first audio stream of the first adaptation set is requested and / or received at a higher bit rate than the bit rate of the second audio stream of the second adaptation set.

15. The system of claim 1, wherein: The at least one first audio element is provided in the first adaptation set, and the at least one second audio element is provided in the second adaptation set, and the at least one third audio element is provided in a third adaptation set, and wherein the first audio scene is described by interaction metadata as an audio scene requiring at least the first adaptation set and the second adaptation set, and wherein the second audio scene is described by interaction metadata as an audio scene requiring at least the third adaptation set and the second adaptation set, The system comprises a metadata processor configured to manipulate the interaction metadata to merge the second adaptation set and the third adaptation set into a new single stream when the second audio scene is reproduced.

16. The system of claim 1, wherein: The system comprises a metadata processor configured to manipulate metadata in at least one audio stream before the at least one audio decoder based on current movement data and / or location data of a user.

17. The system according to claim 16, wherein: The metadata processor is configured to enable and / or disable at least one audio element in at least one audio stream of the first adaptation set or the second adaptation set before the at least one audio decoder based on current movement data and / or location data of the user, wherein the metadata processor is configured to, if the system decides not to reproduce at least one audio element in the at least one audio stream of the first adaptation set or the second adaptation set due to movement data and / or position data, disable the at least one audio element before the at least one audio decoder, and The metadata processor is configured to: enable at least one audio element before the at least one audio decoder when the system decides to reproduce at least one audio element in the at least one audio stream of the first adaptation set or the second adaptation set due to the user's current movement data and / or location data.

18. The system of claim 1, configured to disable decoding of audio elements selected based on current movement data and / or location data of a user.

19. The system of claim 1, configured to merge a first audio stream associated with a current audio scene as the first representation of the first adaptation set with a second audio stream associated with a neighboring audio scene as the second representation of the second adaptation set.

20. The system of claim 1, further configured to: manipulating metadata associated with a group of the first audio stream as the first representation of the first adaptation set and the second audio stream as the second representation of the second adaptation set based on at least current movement data and / or location data of a user to: selecting and activating audio elements that synthesize the audio scene to be reproduced; and All selected audio streams that are selected representations of the first adaptation set and the second adaptation set are merged into a single audio stream or one single audio stream.

21. The system of claim 1 , configured to select between reproducing one audio scene and synthesizing or multiplexing or superimposing or combining at least two audio scenes based on current or future or viewport and movement data and / or position data and / or user selection.

22. The system of claim 1, configured to use at least the first adaptation set and the second adaptation set such that: Multiple adaptation sets are associated with one audio scene; and / or Provides additional information including the following: - information about the boundaries of an audio scene, and / or - information about the relationship between an adaptation set and an audio scene, and / or - Information about connections between boundaries of the audio scene and the plurality of adaptation sets.

23. The system according to claim 1, configured to: receiving the first audio stream as the first representation of the first adaptation set for an audio scene associated with an adjacent or neighboring audio scene; Upon detecting a transition of a boundary between two audio scenes, decoding of the second audio stream as the second representation of the second adaptation set for the adjacent or neighboring audio scene is started.

24. The system of claim 1, further configured to: requesting and / or receiving at least one first representation of the first adaptation set, the at least one first representation comprising a first audio stream associated with at least one first audio scene; requesting and / or receiving a second representation of the second adaptation set, the second representation comprising at least one second audio stream associated with at least two audio scenes, the at least two audio scenes including the at least one first audio scene; as well as The first audio stream and the second audio stream are merged into a new audio stream to be decoded based on available metadata related to the following items: the user's current movement data and / or virtual position data and / or information describing the association of the first adaptation set with the at least one first audio scene and / or the association of the second adaptation set with the at least one first audio scene.

25. The system of claim 1, configured to: receiving information about the user's current movement data and / or location data; and Information about the availability of adaptation sets and information describing an association of at least one adaptation set with at least one audio scene position data and / or movement data is received.

26. The system of claim 1, configured to: deciding whether to reproduce at least one audio element from at least one audio scene embedded in at least one audio stream and at least one additional audio element from at least one additional audio scene embedded in at least one additional audio stream; and In case of a positive decision, an operation is caused of merging or compositing or multiplexing or superimposing or combining the at least one additional audio stream of the additional audio scene to the at least one audio stream of the at least one audio scene.

27. The system of claim 1, configured to: manipulating audio metadata associated with a selected audio stream as a selected representation of the first adaptation set and the second adaptation set based on at least a user's current viewport and / or head orientation and / or movement data and / or metadata and / or virtual position data to: Selecting and / or enabling and / or activating the audio elements that synthesize the audio scene to be reproduced; and All selected audio streams that are selected representations of the first adaptation set and the second adaptation set are enabled to be merged into a single audio stream.

28. The system of claim 1, wherein the interaction metadata further indicates that the second audio element is to be used in an additional audio scene.

29. A method for receiving an audio stream to be reproduced in a media consumption device, comprising: decoding the audio signal from the audio stream to reproduce the audio scene, and requesting and / or receiving a first audio stream as a first representation of a first adaptation set and a second audio stream as a second representation of a second adaptation set based at least on position data and / or movement data of a user, wherein a first audio element in the first audio stream is more relevant and / or more audible than a second audio element in the second audio stream, wherein the first audio stream is the first representation having a higher bit rate than the second audio stream being the second representation, wherein at least one first audio element of the first audio elements is associated only with a first audio scene, and at least one second audio element of the second audio elements is associated only with the first audio scene, and at least one third audio element is associated only with a second audio scene, and The method further comprises: obtaining interaction metadata describing that the at least one second audio element is additionally associated with the second audio scene, requesting and / or receiving the at least one first audio element and the at least one second audio element in case of reproducing the first audio scene, requesting and / or receiving the at least one second audio element and the at least one third audio element in case of reproducing the second audio scene, and In case of a transition between the first audio scene and the second audio scene, the at least one first audio element, the at least one second audio element and the at least one third audio element are requested and / or received.

30. A storage medium storing a computer program, the computer program comprising instructions that, when executed by a processor, cause the processor to perform the method of claim 29.

Images