Smart access to personalized audio

Abstract

To provide a method, a system, and a storage medium for generating a bitstream that represents an object based audio program.SOLUTION: A bitstream comprises a sequence of containers. A first container in the container sequence includes a plurality of substream entities for a plurality of substreams of an object based audio program, and a presentation section. A method 600 includes the steps of: 601 determining a set of object channels; 602 providing a set of object related metadata for the set of object channels; 603 inserting a first set of object channel frames and a first set of object related metadata frames into a respective set of substream entities of the first container; and 604 inserting presentation data into the presentation section.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] This paper relates to audio signal processing, and more particularly to encoding, decoding and interactive rendering of audio data bitstreams that include audio content and metadata that supports interactive rendering of the audio content. [Background technology]

[0002] Audio encoding and decoding that enables a personalized audio experience typically needs to carry all audio object channels and / or audio speaker channels potentially needed for the personalized audio experience. In particular, the audio data / metadata is typically such that parts not needed for a personalized audio program cannot be easily removed from a bitstream containing such a personalized audio program.

[0003] Typically, the entire data for an audio program (audio data and metadata) is stored jointly in the bitstream. The receiver / decoder needs to parse at least the complete metadata to understand which parts of the bitstream (e.g. which speaker channels and / or which object channels) are needed for the personalized audio program. In addition, it is typically not possible without significant computational effort to strip out parts of the bitstream that are not needed for the personalized audio program. In particular, it may be required that parts of the bitstream that are not needed for a given playback scenario / for a given personalized audio program need to be decoded. It may then be required to mute these parts of the bitstream during playback in order to generate the personalized audio program. Furthermore, it may not be possible to efficiently generate a sub-bitstream from the bitstream, where the sub-bitstream contains only the data needed for the personalized audio program. Summary of the Invention [Problem to be solved by the invention]

[0004] This paper addresses the technical problem of providing a bitstream for an audio program that enables a decoder of the bitstream to derive a personalized audio program from the bitstream in a resource-efficient manner. [Means for solving the problem]

[0005] According to one aspect, a method of generating a bitstream indicative of an object-based audio program is described. The bitstream includes a sequence of containers for a corresponding sequence of audio program frames of the object-based audio program. A first container of the sequence of containers includes a plurality of substream entities for a plurality of substreams of the object-based audio program. Further, the first container includes a presentation section. The method includes determining a set of object channels indicative of audio content of at least some of the audio signals of a set of audio signals, where the set of object channels includes a sequence of sets of object channel frames. The method also includes providing or determining a set of object relation metadata for the set of object channels, where the set of object relation metadata includes a sequence of sets of object relation metadata frames. A first audio program frame of the object-based audio program includes a first set of object channel frames of the set of object channel frames and a corresponding first set of object relation metadata frames. The method further includes inserting the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel substream entities of the plurality of substream entities of the first container. Additionally, the method includes inserting presentation data into the presentation section, where the presentation data indicates at least one presentation, the presentation including a set of substream entities from the plurality of substream entities to be presented simultaneously.

[0006] According to another aspect, a bitstream indicative of an object-based audio program is described. The bitstream includes a sequence of containers for a corresponding sequence of audio program frames of the object-based audio program. A first container of the sequence of containers includes a first audio program frame of the object-based audio program. The first audio program frame includes a first set of object channel frames of a set of object channel frames and a corresponding first set of object relation metadata frames. The set of object channel frames indicates audio content of at least some audio signals of a set of audio signals. The first container includes a plurality of substream entities for a plurality of substreams of the object-based audio program. The plurality of substream entities includes a set of object channel substream entities, each for the first set of object channel frames. The first container further includes a presentation section having presentation data, where the presentation data indicates at least one presentation of the object-based audio program. The presentation includes a collection of sub-stream entities from the plurality of sub-stream entities that are to be presented simultaneously.

[0007] According to another aspect, a method for generating a personalized audio program from a bitstream as outlined herein is described. The method includes extracting presentation data from the presentation section, where the presentation data indicates a presentation for a personalized audio program, the presentation including a set of sub-stream entities from the plurality of sub-stream entities to be presented simultaneously. The method further includes extracting one or more object channel frames and corresponding one or more object relations metadata frames from the set of object channel sub-stream entities of the first container based on the presentation data.

[0008] According to a further aspect, a system (e.g., an encoder) is described for generating a bitstream indicative of an object based audio program. The bitstream includes a sequence of containers for a corresponding sequence of audio program frames of the object based audio program. A first container of the sequence of containers includes a plurality of substream entities for a plurality of substreams of the object based audio program. The first container further includes a presentation section. The system is configured to determine a set of object channels indicative of audio content of at least some of the audio signals of a set of audio signals, where the set of object channels includes a sequence of sets of object channel frames. The system is further configured to determine a set of object relation metadata for the set of object channels, where the set of object relation metadata includes a sequence of sets of object relation metadata frames. A first audio program frame of the object based audio program includes a first set of object channel frames of the set of object channel frames and a corresponding first set of object relation metadata frames. Additionally, the system is configured to insert the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel substream entities of the plurality of substream entities of the first container. Further, the system is configured to insert presentation data into the presentation section, where the presentation data indicates at least one presentation, the at least one presentation including a set of substream entities from the plurality of substream entities to be presented simultaneously.

[0009] According to another aspect, a system is described for generating a personalized audio program from a bitstream including an object-based audio program, the bitstream as described herein. The system includes extracting presentation data from the presentation section, where the presentation data indicates a presentation for the personalized audio program, the presentation including a set of sub-stream entities from the plurality of sub-stream entities to be presented simultaneously. The system is further configured to extract, based on the presentation data, one or more object channel frames and corresponding one or more object relations metadata frames from the set of object channel sub-stream entities of the first container.

[0010] According to a further aspect, a software program is described that may be adapted for execution on a processor such that when executed on the processor it performs the method steps outlined herein.

[0011] According to another aspect, a storage medium is described which may carry a software program for execution on a processor adapted to perform the method steps outlined herein when executed on the processor.

[0012] According to a further aspect, a computer program product is described that may include executable instructions for performing the method steps outlined herein when executed on a computer.

[0013] It should be noted that the methods and systems outlined in this patent application, including preferred embodiments thereof, may be used alone or in combination with other methods and systems disclosed herein. Furthermore, all aspects of the methods and systems outlined in this patent application may be combined in any combination. In particular, the features of the claims may be combined with each other in any manner. [Brief description of the drawings]

[0014] The invention is described below in an exemplary manner with reference to the accompanying drawings, in which: [Figure 1] FIG. 2 is a block diagram of an exemplary audio processing chain. [Diagram 2] 1 is a block diagram of an exemplary audio encoder. [Diagram 3] 1 is a block diagram of an exemplary audio decoder. [Figure 4] A diagram showing example presentation data and example substreams of an audio program. [Diagram 5] FIG. 2 illustrates an exemplary structure of a bitstream containing presentation data. [Figure 6] 1 is a flowchart of an exemplary method for generating a bitstream including presentation data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] As mentioned above, the present document is directed to the technical problem of providing a bitstream for a generic audio program, which enables a decoder of said bitstream to generate a personalized audio program from said bitstream in a resource-efficient manner. In particular, the generation of the personalized audio program should be performed with a relatively low computational complexity. Furthermore, said bitstream containing the generic audio program should exhibit a relatively low bitrate.

[0016] Figure 1 shows a block diagram of an exemplary audio processing chain (also referred to as an audio data processing system). The system includes the following elements coupled as shown: a capture unit 1, a production unit 3 (which includes an encoding subsystem), a delivery subsystem 5, a decoder 7, an object processing subsystem 9, a controller 10, and a rendering subsystem 11. Variations on the illustrated system omit one or more of these elements or include additional audio data processing units. Typically, elements 7, 9, 10, and 11 are included in a playback and / or decoding system (e.g., an end user's home theater system).

[0017] The capture unit 1 is typically configured to generate PCM (time domain) samples containing the audio content and to output the PCM samples, which may represent multiple streams of audio captured by microphones (e.g. at a sporting event or other spectator event). The production unit 3, typically operated by a broadcaster, is configured to accept the PCM samples as input and output an object-based audio program representing the audio content. The program is typically or includes an encoded (e.g. compressed) audio bitstream representing the audio content and presentation data allowing various personalized audio programs to be derived from the bitstream. The data of the encoded bitstream representing the audio content is sometimes referred to herein as "audio data". The object-based audio program output from unit 3 may represent (i.e. include) multiple speaker channels of audio data ("beds" of speaker channels), multiple object channels of audio data and object-related metadata. An audio program may include presentation data that may be used to select different combinations of speaker channels and / or object channels to generate different personalized audio programs (which may also be referred to as different experiences). By way of example, an object-based audio program may include a main mix that includes audio content indicative of a bed of speaker channels, audio content indicative of at least one user-selectable object channel (and optionally at least one other object channel), and object-related metadata associated with each object channel.The program may also include at least one side mix including audio content and / or object-related metadata that indicates at least one other object channel (e.g., at least one user-selectable object channel). The audio program may indicate one or more beds for the speaker channels, or may not indicate any beds. For example, an audio program (or a particular mix / presentation) may indicate two or more beds for the speaker channels (e.g., a neutral crowd noise bed for 5.1 channels, a home team crowd noise bed for 2.0 channels, and an away team crowd noise bed for 2.0 channels), including at least one user-selectable bed (which can be selected using a user interface used for user selection of object channel content or configuration) and a default bed (which is rendered in the absence of user selection of another bed). The default bed may be determined by data indicating the configuration (e.g., initial configuration) of the speaker set of the playback system, and optionally, the user may select another bed to be rendered in place of the default bed.

[0018] The delivery subsystem 5 of FIG. 1 is configured to store and / or transmit (e.g., broadcast) the audio program generated by unit 3. The decoder 7 accepts (receives or reads) the audio program delivered by the delivery subsystem 5 and decodes the program (or one or more accepted elements thereof). The object processing subsystem 9 is coupled to receive (from the decoder 7) the decoded speaker channels, object channels, and object related metadata of the delivered audio program. The subsystem 9 is coupled and configured to output to the rendering subsystem 11 a selected subset of the total set of object channels represented by the audio program and the corresponding object related metadata. The subsystem 9 is configured to pass the decoded speaker channels from the decoder 7 through (to the subsystem 11), typically unchanged.

[0019] The object channel selections performed by subsystem 9 may be determined by user selection(s) (indicated by control data presented to subsystem 9 from controller 10) and / or by rules (e.g., indicating conditions and / or constraints) that subsystem 9 is programmed or otherwise configured to implement. Such rules may be determined by object-relational metadata of the audio program and / or by other data presented to subsystem 9 (e.g., from controller 10 or another external source) (e.g., data indicating the capabilities and organization of the speaker array of the playback system) and / or by pre-configuring (e.g., programming) subsystem 9. Controller 10 may provide the user (via a user interface implemented by controller 10) with a menu or palette of selectable "preset" mixes or presentations of object and "bed" speaker channel content (e.g., displayed on a touch screen). The selectable preset mixes or presentations may be determined by presentation data contained within the audio program and possibly also by rules implemented by subsystem 9 (e.g., rules that subsystem 9 is pre-configured to implement). A user selects among the available mixes / presentations by inputting commands into the controller 10 (eg, by manipulating its touch screen), and in response, the controller 10 presents corresponding control data to the subsystem 9.

[0020] The rendering subsystem 11 of FIG. 1 is configured to render audio content determined by the output of the subsystem 9 for playback by speakers (not shown) of a playback system. The subsystem 11 is configured to map audio content determined by object channels selected by the object processing subsystem 9 (e.g., default objects and / or user-selected objects selected as a result of user interaction with the controller 10) to available speaker channels using rendering parameters (e.g., user-selected and / or default values ​​of spatial position and level) output from the subsystem 9 associated with each selected object. At least some of the rendering parameters may be determined by object-related metadata output from the subsystem 9. The rendering system 11 may also receive a bed of speaker channels passed through by the subsystem 9. Typically, the subsystem 11 is an intelligent mixer and is configured to determine speaker feeds for available speakers. This includes by mapping one or more selected (e.g., selected by default) objects to each of several individual speaker channels and mixing those objects with the "bed" audio content presented by each corresponding speaker channel of the program's speaker channel bed.

[0021] FIG. 2 is a block diagram of a broadcast system configured to generate object-based audio programs (and corresponding video programs) for broadcast. A set of X microphones (X being an integer greater than 0, 1, or 2), including microphones 100, 101, 102, 103 of the system of FIG. 2, are positioned to capture audio content to be included in the audio program, with their outputs coupled to the inputs of an audio console 104. The audio program may include interactive audio content that indicates the atmosphere within or at a spectator event (e.g., a soccer or rugby match, a car or motorcycle race, or another sporting event) and / or commentary on the spectator event. The audio program may include a number of audio objects (including user selectable objects or object sets, and typically also a default set of objects that are rendered in the absence of object selection by the user) and a mix (or "bed") of speaker channels for the audio program. The bed of speaker channels may be a conventional mix (e.g., a 5.1 channel mix) of speaker channels of the type that may be included in a conventional broadcast program that does not include object channels.

[0022] In operation, a subset of the microphones (e.g., microphones 100 and 101, and optionally other microphones whose outputs are coupled to audio console 104) may be a conventional microphone array that captures audio (to be encoded and delivered as a bed of speaker channels). In operation, another subset of the microphones (e.g., microphones 102 and 103, and optionally other microphones whose outputs are coupled to audio console 104) captures audio (e.g., crowd noise and / or other "objects") that is to be encoded and delivered as object channels of the program. For example, the microphone array of the system of FIG. 2 may include at least one microphone (e.g., microphone 100) implemented as a sound field microphone and permanently installed in the stadium; at least one stereo microphone (e.g., microphone 102 implemented as a Sennheiser MKH416 microphone or other stereo microphone) pointed toward a position of spectators supporting one team (e.g., the home team), and at least one other stereo microphone (e.g., microphone 103 implemented as a Sennheiser MKH416 microphone or other stereo microphone) pointed toward a position of spectators supporting the other team (e.g., the traveling team).

[0023] The broadcast system of FIG. 2 may include a mobile unit (which may be a truck, sometimes referred to as a "game truck") located outside the stadium (or other event location). This mobile unit is the initial recipient of audio feeds from microphones within the stadium (or other event location). The game truck generates an object-based audio program (to be broadcast) by encoding audio content from the microphones for delivery as object channels of the audio program, generating corresponding object-relational metadata (e.g., metadata indicating the spatial location where each object should be rendered) and including such metadata in the audio program, and / or by encoding audio content from some of the microphones for delivery as beds of speaker channels of the audio program.

[0024] For example, in the system of Figure 2, the console 104, the object processing subsystem 106 (coupled to an output of the console 104), the embedding subsystem 108 and the contributing encoder 110 may be located within the game track. The object-based audio program generated in the subsystem 106 may be combined (e.g., in the subsystem 108) with video content (e.g., from a camera located within the stadium) to generate a combined audio and video signal. The combined signal is then encoded (e.g., by the encoder 110) to generate an encoded audio / video signal for broadcast (e.g., by the delivery subsystem 5 of Figure 1). It should be understood that a playback system for decoding and rendering such an encoded audio / video signal would include a subsystem (not shown separately) for parsing the audio and video content of the delivered audio / video signal, a subsystem for decoding and rendering the audio content, and another subsystem (not shown separately) for decoding and rendering the video content.

[0025] The audio output of the console 104 may include a 5.1 speaker channel bed (labeled in FIG. 2 as “5.1 Neutral”), e.g., indicative of sounds captured at a sporting event; stereo object channel audio content (labeled “2.0 Home”), indicative of crowd noise, e.g., from fans of the home team attending the event; stereo object channel audio content (labeled “2.0 Away”), indicative of crowd noise, e.g., from fans of the visiting team attending the event; object channel audio content (labeled “1.0 cmm1”), indicative of commentary by an announcer, e.g., from the home team's city; object channel audio content (labeled “1.0 cmm2”), indicative of commentary by an announcer, e.g., from the visiting team's city; and object channel audio content (labeled “1.0 Ball Kick”), indicative of sounds produced by a game ball, e.g., as the ball is hit by a participant at the sporting event.

[0026] The object processing subsystem 106 is configured to organize (e.g., group) the audio streams from the console 104 into object channels (e.g., grouping the left and right audio streams labeled “2.0 away” into a roadshow crowd noise object channel) and / or sets of object channels, generate object-related metadata indicative of the object channels (and / or object channel sets), and encode the object channels (and / or object channel sets), the object-related metadata, and the speaker channel beds (determined from the audio streams from the console 104) as an object-based audio program (e.g., an object-based audio program encoded as an AC-4 bitstream). Alternatively, the encoder 110 may be configured to generate an object-based audio program, which may be encoded, for example, as an AC-4 bitstream. In such a case, the object processing subsystem 106 may focus on generating audio content (e.g., using the Dolby E+ format), while the encoder 110 may focus on generating a bitstream for transmission or distribution.

[0027] Subsystem 106 may further be configured to render (and play on a set of studio monitor speakers) at least a selected subset of the speaker channel bed and object channel (and / or object channel set) (including by using the object relational metadata to generate a mix / presentation indicating the selected object channel(s) and speaker channels), so that the played sound can be monitored by operator(s) of console 104 and subsystem 106 (as indicated by the “monitor path” in FIG. 2 ).

[0028] The interface between the output of subsystem 104 and the input of subsystem 106 may be a Multichannel Audio Digital Interface (MADI).

[0029] In operation, subsystem 108 of the system of Figure 2 combines the object-based audio program generated in subsystem 106 with video content (e.g., from a camera located within the stadium) to generate a combined audio and video signal that is presented to encoder 110. The interface between the output of subsystem 108 and the input of subsystem 110 may be a high-definition serial digital interface (HD-SDI). In operation, encoder 110 encodes the output of subsystem 108, thereby generating an encoded audio / video signal for broadcast (e.g., by delivery subsystem 5 of Figure 1).

[0030] A broadcast facility (e.g., subsystems 106, 108, and 110 of the system of FIG. 2) may be configured to generate various presentations of the elements of an object-based audio program. Examples of such presentations include 5.1 flattened mixes, international mixes, and national mixes. For example, all presentations may include a common bed of speaker channels, but the object channels of the presentations (and / or a menu of selectable object channels and / or selectable or non-selectable rendering parameters for rendering and mixing the object channels as determined by the presentation) may vary from presentation to presentation.

[0031] The object-relationship metadata of an audio program (or pre-configured settings of a playback or rendering system not dictated by metadata delivered with the audio program) may impose constraints or conditions on the selectable mixing / presentation of object and bed (speaker channel) content. For example, a DRM hierarchy may be implemented allowing a user to have tiered access to a set of audio channels contained in an object-based audio program. If a user pays more money (e.g., to the broadcaster), the user may be licensed to decode, select, and render more object channels of that audio program.

[0032] Figure 3 is a block diagram of an exemplary playback system including a decoder 20, an object processing subsystem 22, a spatial rendering subsystem 24, a controller 23 (which implements a user interface), and optionally digital audio processing subsystems 25, 26, and 27 coupled as shown. In some implementations, elements 20, 22, 24, 25, 26, 27, 29, 31, and 33 of the system of Figure 3 are implemented as set-top devices.

[0033] In the system of Figure 3, the decoder 20 is configured to receive and decode an encoded signal indicative of an object-based audio program. The audio program may, for example, indicate audio content including two speaker channels (i.e., a "bed" of at least two speaker channels). The audio program may also indicate at least one user-selectable object channel (and optionally at least one other object channel) and object-related metadata corresponding to each object channel. Each object channel indicates an audio object, and thus the object channels are sometimes referred to herein as "objects" for convenience. The audio program may be contained within an AC-4 bitstream indicating the audio objects, the object-related metadata, and / or the bed of speaker channels. Typically, individual audio objects are mono or stereo coded (i.e. each object channel represents the left or right channel of the object or is a monophonic channel representing the object), the bed may be a traditional 5.1 mix and the decoder 20 may be configured to simultaneously decode a predetermined number (e.g. 16 or more) of channels of audio content (including, for example, six speaker channels for the bed and, for example, 10 or more object channels). The incoming bitstream may represent a number (e.g. more than 10) audio objects, not all of which need to be decoded to achieve a particular mix / presentation.

[0034] As mentioned above, an audio program may include zero, one or more beds of speaker channels as well as one or more object channels. The beds of speaker channels and / or the object channels may form substreams of a bitstream including the audio program. Thus, a bitstream may include multiple substreams, where a substream indicates a bed of speaker channels or one or more object channels. Furthermore, the bitstream may include presentation data (e.g., included within a presentation section of the bitstream), where the presentation data may indicate one or more different presentations. A presentation may define a particular mix of the substreams. In other words, a presentation may define a bed of speaker channels and / or one or more object channels that are to be mixed together to provide a personalized audio program.

[0035] 4 illustrates a number of substreams 411, 412, 413, 414. Each substream 411, 412, 413, 414 includes audio data 421, 424, where the audio data 421, 424 may correspond to a bed of speaker channels or to audio data of an audio object (i.e., to an audio channel). As an example, the substream 411 may include the bed of speaker channels 421 and the substream 414 may include the object channel 424. Additionally, each substream 411, 412, 413, 414 may include metadata 431, 434 (e.g., default metadata) that is associated with the audio data 421, 424 and that may be used to render the associated audio data 421, 424. As an example, sub-stream 411 may contain speaker-related metadata (for the bed in speaker channel 421) and sub-stream 414 may contain object-related metadata (for object channel 424). In addition, sub-streams 411, 412, 413, 414 may contain alternative metadata 441, 444 to provide one or more alternative ways of rendering the associated audio data 421, 424.

[0036] 4 further illustrates different presentations 401, 402, 403. Presentation 401 indicates a selection of substreams 411, 412, 413, 414 to be used for presentation 401, thereby defining a personalized audio program. Presentation 401 may further indicate metadata 431, 441 (e.g., one of default metadata 431 or alternative metadata 441) to be used for the substream 411 selected for presentation 401. In the illustrated example, presentation 401 describes a personalized audio program including substreams 411, 412, 414.

[0037] Thus, the use of the presentations 401, 402, 403 provides an efficient means of signaling different personalized audio programs within a general object-based audio program. In particular, the presentations 401, 402, 403 may be such that the decoder 7, 20 can simply select the one or more substreams 411, 412, 413, 414 required for a particular presentation 401 without having to decode the complete bitstream of the general object-based audio program. For example, a re-multiplexer (not shown in Fig. 3) may be configured to simply extract the one or more substreams 411, 412, 413, 414 from the complete bitstream to generate a new bitstream for the personalized audio program of the particular presentation 401. In other words, from a bitstream with a relatively large number of presentations 401, 402, 403, a new bitstream carrying a reduced number of presentations may be efficiently generated. A possible scenario is a relatively large bitstream with a relatively large number of presentations arriving at an STB. The STB may be adapted to focus on personalization (i.e., selecting the presentation) and may be configured to repackage the single presentation bitstream (without decoding the audio data). The single presentation bitstream (and audio data) may then be decoded in a suitable remote decoder, for example in an AVR (audio / video receiver) or in a mobile home device such as a tablet PC.

[0038] A decoder (e.g., decoder 20 of FIG. 3) may parse the presentation data to identify presentation 401 for rendering. Additionally, decoder 200 may extract sub-streams 411, 412, 414 needed for presentation 401 from locations indicated by the presentation data. After extracting sub-streams 411, 412, 414 (speaker channels, object channels, and associated metadata), the decoder may perform any necessary decoding on (e.g., only on) the extracted sub-streams 411, 412, 414.

[0039] The bitstream may be an AC-4 bitstream and the presentations 401, 402, 403 may be AC-4 presentations. These presentations allow easy access to the parts of the bitstream (audio data 421 and metadata 431) that are needed for a particular presentation. In that way, the decoder or receiver system 20 can easily access the needed parts of the bitstream without having to parse deeply into other parts of the bitstream. This allows for example the possibility to transfer only the needed parts of the bitstream to another device without having to reconstruct the entire structure or even to decode and encode the substreams 411, 412, 413, 414 of the bitstream. In particular, a reduced structure derived from the bitstream may be extracted.

[0040] Referring again to FIG. 3, a user may use controller 23 to select an object (represented by an object-based audio program) to be rendered. As an example, the user may select a particular presentation 401. Controller 23 may be a handheld processing device (e.g., an iPad®) that is programmed to implement a user interface (e.g., an iPad® app) that is compatible with other elements of the system of FIG. 3. The user interface may provide the user (e.g., displayed on a touch screen) with a menu or palette of selectable presentations 401, 402, 403 (e.g., “preset” mixes) of the object and / or “bed” speaker channel content. Presentations 401, 402, 403 may be provided with name tags in the menu or palette. The selectable presentations 401, 402, 403 may be determined by the presentation data in the bitstream and possibly also by rules implemented by subsystem 22 (e.g., rules that subsystem 22 is preconfigured to implement). A user may select from among the available presentations by inputting a command to controller 23 (e.g., by activating a touch screen of controller 23), and in response, controller 23 may present corresponding control data to subsystem 22.

[0041] In response to the object-based audio program, and in response to control data from controller 23 indicating the selected presentation 401, decoder 20 decodes (if necessary) the speaker channels of the selected presentation 401 and outputs the decoded speaker channels to subsystem 22. In response to the object-based audio program, and in response to control data from controller 23 indicating the selected presentation 401, decoder 20 decodes (if necessary) the selected object channels and outputs the selected (e.g., decoded) object channels (each of which may be a pulse code modulated or "PCM" bitstream) and object related metadata corresponding to the selected object channels to subsystem 22.

[0042] The objects represented by the decoded object channels typically are or include user-selectable audio objects. For example, as shown in Figure 3, the decoder 20 may include a 5.1 speaker channel bed, an object channel showing commentary by an announcer from the home team's city ("Commentary 1 Mono"), an object channel showing commentary by an announcer from the visiting team's city ("Commentary 2 Mono"), a stereo object channel showing crowd noise from the home team's fans attending the sporting event ("Fans (Home)"), left and right object channels showing sounds made by the game ball as it is hit by a sporting event participant ("Ball Sound Stereo"), and four object channels showing special effects ("Effects 4x Mono"). Any of the “Comment 1 Mono”, “Comment 2 Mono”, “Fan (Home)”, “Ball Sound Stereo” and “Effects 4x Mono” object channels may be selected as part of the presentation 401, and each one selected is passed from subsystem 22 (after any necessary decoding in decoder 20) to the rendering subsystem 24.

[0043] Subsystem 22 is configured to output a selected subset of the full set of object channels presented by an audio program and the corresponding object-related metadata of the audio program. The object selection may be determined by user selection (indicated by control data presented to subsystem 22 from controller 23) and / or by rules (e.g., indicating conditions and / or constraints) that subsystem 22 has been programmed or otherwise configured to implement. Such rules may be determined by the object-related metadata of the program and / or by other data presented to subsystem 22 (e.g., from controller 23 or another external source) (data indicating the capabilities and organization of the speaker array of the playback system) and / or by pre-configuring (e.g., programming) subsystem 22. As noted above, the bitstream may include presentation data that provides a set of selectable "preset" mixes (i.e., presentations 401, 402, 403) of object and "bed" speaker channel content. Subsystem 22 passes the decoded speaker channels from decoder 20, typically unchanged (to subsystem 24), and processes the selections of object channels presented to it.

[0044] 3 (or subsystem 24 together with at least one downstream device or system) is configured to render the audio content output from subsystem 22 for playback through speakers of a user's playback system. One or more of the optionally included digital audio processing subsystems 25, 26, and 27 may implement post-processing on the output of subsystem 24.

[0045] The spatial rendering subsystem 24 is configured to map the audio object channels selected by the object processing subsystem 22 to the available speaker channels using the rendering parameters (e.g., user-selected and / or default values ​​of spatial position and level) output from subsystem 22 associated with each selected object. The spatial rendering subsystem 24 also receives the bed of decoded speaker channels passed through by subsystem 22. Typically, subsystem 24 is an intelligent mixer and is configured to map one, two or more selected object channels to each of several individual speaker channels and determine the speaker feeds for the available speakers, including by mixing the selected object channel(s) with the "bed" audio content presented by each corresponding speaker channel of the program's speaker channel bed.

[0046] The speakers driven to render the audio may be located at any position in the playback environment, not just in a (nominally) horizontal plane. In some such cases, metadata included with the program indicates rendering parameters for rendering at least one object of the program at any apparent spatial position (in a three-dimensional volume) using a three-dimensional array of speakers. For example, an object channel may have corresponding metadata that indicates a three-dimensional locus of apparent spatial positions at which an object (represented by that object channel) should be rendered. The locus may include a sequence of "floor" positions (in the plane of a subset of speakers assumed to be located on the floor or another horizontal plane of the playback environment) and a sequence of "above-floor" positions (each determined by driving a subset of speakers assumed to be located in at least one other horizontal plane of the playback environment). In such a case, rendering may be performed such that, according to the invention, the speakers may be driven to emit sounds perceived as emanating from a sequence of object positions in three-dimensional space including said trajectory (sounds determined by the associated object channels) mixed with sounds determined by the "bed" audio content. Subsystem 24 may be configured to implement such rendering, or steps thereof, and the remaining steps of the rendering may be performed by a downstream system or device (e.g., rendering subsystem 35 of FIG. 3).

[0047] Optionally, digital audio processing (DAP) stages (e.g., one for each of several predetermined output speaker channel configurations) are coupled to the output of the spatial rendering subsystem 24 to perform post-processing on the output of the spatial rendering subsystem. Examples of such processing include intelligent equalization or (in the case of a stereo output) speaker virtualization processing.

[0048] The output of the system of FIG. 3 (e.g., the output of a spatial rendering subsystem or a DAP stage following the spatial rendering stage) may be a PCM bitstream that determines the speaker feeds for the available speakers. For example, if a user's playback system includes a 7.1 array of speakers, the system may output a PCM bitstream (generated in subsystem 24) that determines the speaker feeds for the speakers in such an array, or a post-processed version of such a bitstream (generated in DAP 25). As another example, if a user's playback system includes a 5.1 array of speakers, the system may output a PCM bitstream (generated in subsystem 24) that determines the speaker feeds for the speakers in such an array, or a post-processed version of such a bitstream (generated in DAP 26). As another example, if a user's playback system only includes left and right speakers, the system may output a PCM bitstream (generated in subsystem 24) that determines the speaker feeds for the left and right speakers, or a post-processed version of such a bitstream (generated in DAP 27).

[0049] 3 also optionally includes one or both of re-encoding subsystems 31 and 33. Re-encoding subsystem 31 is configured to re-encode the PCM bitstream (representing a feed for a 7.1 speaker array) output from DAP 25 as an encoded bitstream (e.g., an AC-4 or AC-3 bitstream), and the resulting encoded (compressed) AC-3 bitstream may be output from the system. Re-encoding subsystem 33 is configured to re-encode the PCM bitstream (representing a feed for a 5.1 speaker array) output from DAP 27 as an encoded bitstream (e.g., an AC-4 or AC-3 bitstream), and the resulting encoded (compressed) bitstream may be output from the system.

[0050] 3 also optionally includes a re-encoding (or formatting) subsystem 29 and a downstream rendering subsystem 35 coupled to receive the output of subsystem 29. Subsystem 29 is coupled to receive data (output from subsystem 22) indicative of selected audio objects (or a default mix of audio objects), corresponding object-related metadata, and bed of speaker channels, and is configured to re-encode (and / or format) such data for rendering by subsystem 35. Subsystem 35 may be implemented in an AVR or soundbar (or other system or device downstream from subsystem 29) and is configured to generate speaker feeds (or a bitstream determining speaker feeds) for available playback speakers (speaker array 36) in response to the output of subsystem 29. For example, subsystem 29 may be configured to generate encoded audio by re-encoding the data indicative of selected (or default) audio objects, corresponding metadata, and bed of speaker channels into a suitable format for rendering in subsystem 35, and transmit the encoded audio to subsystem 35 (e.g., via an HDMI link). In response to the speaker feeds generated by (or determined by) subsystem 35, available speakers 36 emit sounds indicative of the speaker channel bed and a mix of the selected (or default) object(s) with the object(s) whose apparent source location is determined by the object relational metadata in the output of subsystem 29. When subsystems 29 and 35 are included, rendering subsystem 24 is optionally omitted from the system.

[0051] As mentioned above, the use of presentation data is beneficial as it allows the decoder 20 to efficiently select the sub-stream(s) 411, 412, 413, 414 required for a particular presentation 401. In view of this, the decoder 20 may be configured to extract the sub-stream(s) 411, 412, 413, 414 of a particular presentation 401 and to reconstruct a new bit-stream comprising (typically only) the sub-stream(s) 411, 412, 413, 414 of the particular presentation 401. This extraction and reconstruction of the new bit-stream may be performed without the need to actually decode and re-encode said sub-stream(s) 411, 412, 413, 414. Thus, the generation of the new bit-stream for a particular presentation 401 may be performed in a resource-efficient manner.

[0052] The system of Figure 3 may be a distributed system for rendering object-based audio in which a portion (i.e., at least one step) of the rendering (e.g., selection of audio objects to be rendered and selection of rendering characteristics of each selected object, as performed by subsystem 22 and controller 23 of the system of Figure 3) is implemented in a first subsystem (e.g., elements 20, 22, and 23 of Figure 3 implemented in a set-top device or a set-top device and a handheld controller) and another portion of the rendering (e.g., immersive rendering in which speaker feeds or signals determining speaker feeds are generated in response to the output of the first subsystem) is implemented in a second subsystem (e.g., subsystem 35 implemented in an AVR or sound bar). Latency management may be implemented to account for the different times and different subsystems at which portions of the audio rendering (and any processing of the video corresponding to the rendered audio) are performed.

[0053] As shown in FIG. 5, a generic audio program may be transported in a bitstream 500 including a sequence of containers 501. Each container 501 may contain audio program data for a particular frame of the audio program. A particular frame of the audio program may correspond to a particular time segment of the audio program (e.g., 20 milliseconds of the audio program). Thus, each container 501 of the sequence of containers 501 may carry data for a frame of the sequence of frames of the generic audio program. The data for the frame may be included in a frame entity 502 of the container 501. The frame entity may be identified using a syntax element of the bitstream 500.

[0054] As mentioned above, the bitstream 500 may carry a number of substreams 411, 412, 413, 414, where each substream 411 includes a speaker channel bed 421 or an object channel 424. Thus, the frame entity 502 may include a number of corresponding substream entities 520. Furthermore, the frame entity 502 may include a presentation section 510 (also referred to as a Table of Contents (TOC) section). The presentation section 510 may include TOC data 511, which may for example indicate several presentations 401, 402, 403 included within the presentation section 510. Furthermore, the presentation section 510 may include one or more presentation entities 512, each carrying data for defining one or more presentations 401, 402, 403. The substream entity 520 may include a content subentity 521 for carrying audio data 421, 424 of a frame of the substream 411. Furthermore, the sub-stream entity 520 may include a metadata sub-entity 522 for carrying corresponding metadata 431 , 441 for the frames of the sub-stream 411 .

[0055] FIG. 6 shows a flow chart of an exemplary method 600 for generating a bitstream 500 representing an object-based audio program (i.e. a general audio program). The bitstream 500 shows a bitstream format such that the bitstream 500 includes a sequence of containers 501 for a corresponding sequence of audio program frames of the object-based audio program. In other words, each frame (i.e. each temporal segment) of the object-based audio program may be inserted into a container of a sequence of containers that may be defined by the bitstream format. The containers may be defined using specific container syntax elements of the bitstream format. As an example, the bitstream format may correspond to the AC-4 bitstream format. In other words, the bitstream 500 to be generated may be an AC-4 bitstream.

[0056] Furthermore, the bitstream format may be such that a first container 501 of the sequence of containers 501 (i.e. at least one of the containers 501 of the sequence of containers 501) includes multiple substream entities 520 for multiple substreams 411, 412, 413, 414 of an object-based audio program. As outlined above, an audio program may include multiple substreams 411, 412, 413, 414, and each substream 411, 412, 413, 414 may include a bed of speaker channels 421 or an object channel 424 or both. The bitstream format may be such that each container 501 of the sequence of containers 501 provides a dedicated substream entity 520 for the corresponding substream 411, 412, 413, 414. In particular, each substream entity 520 may include data relating to frames of the corresponding substream 411, 412, 413, 414. The frames of the substreams 411, 412, 413, 414 may be frames of a speaker channel 421, referred to herein as speaker channel frames. Alternatively, the frames of the substreams 411, 412, 413, 414 may be frames of an object channel, referred to herein as object channel frames. The substream entities 520 may be defined by corresponding syntax elements of the bitstream format.

[0057] Furthermore, the first container 501 may include a presentation section 510. In other words, the bitstream format may allow the definition (e.g. using appropriate syntax elements) of a presentation section 510 for every container 501 of a sequence of containers 501. The presentation section 510 may be used to define different presentations 401, 402, 403 for different personalized audio programs that can be generated from a (generic) object-based audio program.

[0058] The method 600 includes determining 601 a set of object channels 424 indicative of audio content of at least some of the set of audio signals. The set of audio signals may represent captured audio content, for example audio content captured using the system described in the context of FIG. 2. The set of object channels 424 may include multiple object channels 424. Furthermore, the set of object channels 424 includes a sequence of a set of object channel frames. In other words, each object channel includes a sequence of object channel frames. As a result, the set of object channels includes a sequence of a set of object channel frames, and the set of object channel frames at a particular time point includes the object channel frames of the set of object channels at that particular time point.

[0059] Further, the method 600 includes providing or determining 602 a set of object relation metadata 434, 444 for a set of object channels 424, where the set of object relation metadata 434, 444 includes a sequence of sets of object relation metadata frames. In other words, the object relation metadata for an object channel is segmented into a sequence of object relation metadata frames. As a result, the set of object relation metadata for a corresponding set of object channels includes a sequence of sets of object relation metadata frames.

[0060] Thus, an object related metadata frame may be provided (e.g., using the object processor 106 described in the context of FIG. 2) for the corresponding object channel frame. As mentioned above, the object channel 424 may be provided with different variants of the object related metadata 434, 444. By way of example, a default variant of the object related metadata 434 and one or more alternative variants of the object related metadata 444 may be provided. By doing so, different perspectives (e.g., different locations in a stadium) may be simulated. Alternatively or additionally, the speaker channel bed 421 may be provided with different variants of the speaker related metadata 431, 441. By way of example, a default variant of the speaker related metadata 431 and one or more alternative variants of the speaker related metadata 441 may be provided. By doing so, different rotations of the speaker channel bed 421 may be defined. As with the object related metadata, the speaker related metadata may also be time-varying.

[0061] Thus, an audio program may have a set of object channels such that a first audio program frame of an object-based audio program includes a first set of object channel frames from the sequence of sets of object channel frames and a corresponding first set of object relation metadata frames from the sequence of sets of object relation metadata frames.

[0062] The method 600 further includes inserting 603 the first set of object channel frames and the first set of object related metadata frames into a respective set of object channel substream entities 520 of the plurality of substream entities 520 of the first container 501. Thus, for each object channel 421 of an object-based audio program, a substream 411, 412, 413, 414 may be generated. Each substream 411, 412, 413, 414 may be identified in the bitstream 500 via the respective substream entity 520 carrying the substream 411, 412, 413, 414. As a result of this, the various substreams 411, 412, 413, 414 may be identified and potentially extracted by the decoder 7, 20 in a resource-efficient manner without the need to decode the complete bitstream 500 and / or the substreams 411, 412, 413, 414.

[0063] Further, the method 600 includes inserting 604 presentation data into the presentation section 510 of the bitstream 500. The presentation data may indicate at least one presentation 401, where the at least one presentation 401 may define a personalized audio program. In particular, the at least one presentation 401 may include or indicate a collection of substream entities 520 from the plurality of substream entities 520 to be presented simultaneously. Thus, the presentation 401 may indicate which one or more of the substreams 411, 412, 413, 414 of the object-based audio program are selected to generate a personalized audio program. As outlined above, the presentation 401 may identify a subset of the complete collection of substreams 411, 412, 413, 414 (i.e., less than the total number of substreams 411, 412, 413, 414).

[0064] The insertion of presentation data enables a corresponding decoder 7, 20 to identify and extract one or more substreams 411, 412, 413, 414 from the bitstream 500 to generate a personalized audio program without having to decode or parse the complete bitstream 500.

[0065] The method 600 may include determining a bed of speaker channels 421 indicative of audio content of one or more audio signals of the set of audio signals. The bed of speaker channels 421 may include one or more of: 2.0 channels, 5.1 channels, 5.1.2 channels, 7.1 channels, and / or 7.1.4 channels. The bed of speaker channels 421 may be used to provide a basis for a personalized audio program. Additionally, one or more object channels 424 may be used to provide a personalized variation of the personalized audio program.

[0066] The speaker channel bed 421 may include a sequence of speaker channel frames, and a first audio program frame of an object-based audio program may include a first speaker channel frame of a sequence of speaker channel frames. The method 600 may further include inserting the first speaker channel frame into a speaker channel substream entity 520 of the plurality of substream entities 520 of the first container 501. The presentation 401 in the presentation section 510 may then include or indicate the speaker channel substream entity 520. Alternatively or additionally, the presentation 401 may include or indicate one or more object channel substream entities 520 from a set of object channel substream entities.

[0067] The method 600 may further include providing speaker related metadata 431, 441 for the speaker channel bed 421. The speaker related metadata 431, 441 may include a sequence of speaker related metadata frames. A first speaker related metadata frame from the sequence of speaker related metadata frames may be inserted into the speaker channel sub-stream entity 520. It should be noted that multiple speaker channel beds 421 may be inserted into multiple corresponding speaker channel sub-stream entities 520.

[0068] As outlined in the context of Fig. 4, the presentation data may show multiple presentations 401, 402, 403 including different collections of sub-stream entities 520 for different personalized audio programs. The different collections of sub-stream entities 520 may include the one or more speaker channel sub-stream entities 520, different combinations of the one or more object channel sub-stream entities 520 and / or different combinations of metadata variants 434, 444 (e.g. default metadata 434 or alternative metadata 444).

[0069] The presentation data in the presentation section 510 may be segmented (e.g. using appropriate syntax elements of a bitstream format) into different presentation data entities 512 for the different presentations 401, 402, 403. The method 600 may further include inserting table of contents (TOC) data into the presentation section 510. The TOC data may indicate the locations of the various presentation data entities 512 within the presentation section 510 and / or identifiers for the various presentations 401, 402, 403 contained within the presentation section 510. Thus, the TOC data may be used by a corresponding decoder 7, 20 to identify and extract the various presentations 401, 402, 403 in an efficient manner. Alternatively or additionally, the presentation data entities 512 for the various presentations 401, 402, 403 may be included sequentially within the presentation section 510. If the TOC does not indicate the location of the various presentation data entities 512, the corresponding decoder 7, 20 may identify and extract the various presentations 401, 402, 403 by sequentially parsing through the various presentation data entities 512. This may be a bitrate efficient way to signal the various presentations 401, 402, 403.

[0070] The substream entity 520 may include a content subentity 521 for the audio content or audio data 424 and a metadata subentity 522 for the associated metadata 434, 444. The subentities 521, 522 may be identified by appropriate syntax elements of the bitstream format, so that a corresponding decoder 7, 20 may identify the audio data and corresponding metadata of the object channel or of the speaker channel in a resource-efficient manner.

[0071] As already mentioned above, a metadata frame for a corresponding channel frame may contain multiple different variants or groups of metadata 434, 444. The presentation 401 may indicate which variant or group of metadata 434 should be used to render the corresponding channel frame, thereby allowing for an increased degree of personalization of the audio program (e.g. listening / viewing perspective).

[0072] The speaker channel bed 421 typically includes one or more speaker channels to be respectively presented by one or more speakers 36 of the presentation environment. The object channel 424, on the other hand, is typically presented by a combination of speakers 36 of the presentation environment. The object relation metadata 434, 444 of the object channel 424 may indicate a location from which the object channel 424 should be rendered in the presentation environment. The location of the object channel 424 may vary in time. As a result of this, the combination of speakers 36 for rendering the object channel 424 may vary along the sequence of object channel frames of the object channel 424 and / or the panning of the speakers 36 of the speaker combination may vary along the sequence of object channel frames of the object channel 424.

[0073] The presentations 401, 402, 403 may include target device configuration data for the target device configuration. In other words, the presentations 401, 402, 403 may depend on the target device configuration used for rendering the presentations 401, 402, 403. The target device configurations may differ in terms of the number of speakers, the positions of the speakers, and / or in terms of the number of audio channels that can be processed and rendered. Exemplary target device configurations include a 2.0 (stereo) target device configuration with left and right speakers or a 5.1 target device configuration. The target device configurations typically include the spatial rendering subsystem 24 described in the context of FIG. 3.

[0074] Thus, presentations 401, 402, 403 may indicate different audio resources to be used for different target device configurations. The target device configuration data may indicate a set of substream entities 520 and / or metadata transformations 434 from the plurality of substream entities 520 to be used to render presentation 401 on a particular target device configuration. In particular, the target device configuration data may indicate such information for a plurality of different target device configurations. By way of example, presentation 401 may include different sections with target device configuration data for different target device configurations.

[0075] By doing this, a corresponding decoder or demultiplexer can efficiently identify the audio resources (one or more substreams 411, 412, 413, 414, one or more variants of metadata 441) that should be used for a particular target device configuration.

[0076] The bitstream format may allow further (intermediate) layers for defining personalized audio programs. In particular, the bitstream format may allow the definition of a substream group comprising one, two or more of the plurality of substreams 411, 412, 413, 414. A substream group may be used to group different audio contents such as atmosphere content, dialogue and / or effects. The presentation 401 may show a substream group. In other words, the presentation 401 may identify one, two or more substreams to be rendered simultaneously by referring to a substream group comprising the one, two or more substreams. Thus, the substream group provides an efficient means for identifying two or more substreams (possibly associated with each other).

[0077] The presentation section 510 may include one or more substream group entities (not shown in FIG. 5) for defining one or more corresponding substream groups. The substream group entities may be located after or downstream of the presentation data entity 512. The substream group entities may indicate one or more substreams 411, 412, 413, 414 contained within the corresponding substream group. The presentation 401 (defined within the corresponding presentation data entity 512) may indicate a substream group entity to include the corresponding substream group in the presentation 401. The decoder 7, 20 may parse through the presentation data entities 512 to identify a particular presentation 401. If the presentation 401 references a substream group or a substream group entity, the decoder 7, 20 may continue parsing through the presentation section 510 to identify the definition of the substream group contained within the substream group entity of the presentation section 510. Thus, the decoder 7 , 20 may determine the sub-streams 411 , 412 , 413 , 414 for a particular presentation 401 by parsing through the presentation data entities 512 and through the sub-stream group entities of the presentation section 510 .

[0078] Thus, the method 600 of generating the bitstream 500 may include inserting data identifying the one, two or more of the plurality of substreams into a substream group entity of the presentation section 510. As a result, the substream group entity includes data for defining a substream group.

[0079] The definition of substream groups may be beneficial in terms of bitrate reduction. In particular, multiple substreams 411, 412, 413, 414 that are used jointly in multiple presentations 401, 402, 403 may be grouped into a substream group. As a result, the multiple substreams 411, 412, 413, 414 may be efficiently identified in the presentations 401, 402, 403 by referencing the substream group. Furthermore, the definition of substream groups may provide an efficient means for a content designer to master combinations of the substreams 411, 412, 413, 414 and to define substream groups for the mastered combinations of the substreams 411, 412, 413, 414.

[0080] Thus, a bitstream 500 is described that indicates an object-based audio program and allows for resource-efficient personalization. The bitstream 500 includes a sequence of containers 501 for a corresponding sequence of audio program frames of the object-based audio program. A first container 501 of the sequence of containers 501 includes a first audio program frame of the object-based audio program. The first audio program frame includes a first set of object channel frames of a set of object channels and a corresponding first set of object related metadata frames. The set of object channels may indicate audio content of at least some audio signals of a set of audio signals. Furthermore, the first container 501 includes a plurality of substream entities 520 for a plurality of substreams 411, 412, 413, 414 of the object-based audio program. The plurality of substream entities 520 include a set of object channel substream entities 520 for the first set of object channel frames, respectively. The first container 501 further includes a presentation section 510 with presentation data, where the presentation data may indicate at least one presentation 401 of an object-based audio program, the at least one presentation 401 including a collection of sub-stream entities 520 from the plurality of sub-stream entities 520 to be presented simultaneously.

[0081] The first audio program frame may further include a first speaker channel frame of a bed of speaker channels 421, where the bed of speaker channels 421 indicates audio content of one or more audio signals of the set of audio signals. The plurality of substream entities 520 of the bitstream 500 may then include a speaker channel substream entity 520 for the first speaker channel frame.

[0082] The bitstream 500 may be received by a decoder 7, 20. The decoder 7, 20 may be configured to perform a method for generating a personalized audio program from the bitstream 500. The method may include extracting presentation data from the presentation section 501. As described above, the presentation data may indicate a presentation 401 for the personalized audio program. The method may further include extracting, based on the presentation data, one or more object channel frames and corresponding one or more object relation metadata frames from the set of object channel substream entities 520 of said first container 501 in order to generate and / or render the personalized audio program. Depending on the content of the bitstream, the method may further include extracting, based on the presentation data, a first speaker channel frame from the speaker channel substream entities 520 of the first container 501.

[0083] The methods and bitstreams described herein are useful in view of the generation of personalized audio programs for general object-based audio programs. In particular, the methods and bitstreams described allow portions of the bitstream to be stripped or extracted in a resource-efficient manner. As an example, if only a portion of the bitstream needs to be transferred, this can be done without transferring / processing the full set of metadata and / or the full set of audio data. Only the required portion of the bitstream needs to be processed and transferred. The decoder may only be required to parse the presentation section of the bitstream (e.g., TOC data) to identify the content contained within the bitstream. Furthermore, the bitstream may provide a "default" presentation (e.g., "standard mix") that can be used by the decoder to begin rendering the program without further parsing. Additionally, the decoder only needs to decode the portions of the bitstream that are required to render a particular personalized audio program. This is achieved by appropriate clustering of the audio data into substreams and substream entities. An audio program may contain a potentially unlimited number of substreams and substream entities, thereby allowing a high degree of flexibility in bitstream formats.

[0084] The methods and systems described herein may be implemented as software, firmware and / or hardware. Certain components may be implemented as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or as application specific integrated circuits. Signals encountered in the methods and systems described may be stored on a medium such as a random access memory or optical storage medium, or may be transmitted over a network such as a radio network, satellite network, wireless network or wired network, e.g., the Internet. Typical devices utilizing the methods and systems described herein are portable electronic devices or other consumer equipment used to store and / or render audio signals.

[0085] Embodiments of the invention may relate to one or more of the following enumerated examples (EE). [EEE1] A method (600) for generating a bitstream (500) representing an object-based audio program, the bitstream (500) comprising: a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) of the sequence of containers (501) comprising a plurality of substream entities (520) for a plurality of substreams (411, 412, 413, 414) of the object-based audio program; the first container (501) further comprising a presentation section (510); the method (600) comprising: determining (601) a set of object channels (424) indicative of audio content of at least some of the set of audio signals, the set of object channels (424) comprising a sequence of a set of object channel frames; providing (602) a set of object related metadata (434, 444) for the set of object channels (424), the set of object related metadata (434, 444) comprising a sequence of a set of object related metadata frames; a first audio program frame of the object based audio program comprising a first set of object channel frames and a corresponding first set of object related metadata frames; inserting (603) the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel sub-stream entities (520) of the plurality of sub-stream entities (520) of the first container (501); inserting (604) presentation data into the presentation section (510), the presentation data representing at least one presentation (401); the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously, method. [EEE2] The method (600) of EEE1, wherein the presentation (401) includes one or more object channel substream entities (520) from the set of object channel substream entities. [EEE3] The method (600) of any one of EEE1 or 2, wherein the presentation data indicates a plurality of presentations (401, 402, 403) comprising different collections of sub-stream entities (520), the different collections of sub-stream entities (520) comprising different combinations of object channel sub-stream entities (520) of the collection. [EEE4] The method (600) of any one of EEE1 to 3, wherein the presentation data is segmented into different presentation data entities (512) for different presentations (401, 402, 403). [EEE5] The method further includes inserting table of contents data, referred to as TOC data, into the presentation section (510), the TOC data comprising: the location of the different presentation data entities (512) within the presentation section (510); and / or indicating identifiers for the different presentation data entities (512) contained within the presentation section (510), The method described in EEE4 (600). [EEE6] The method (600) of any one of EEE1 to EEE5, wherein the sub-stream entity (520) includes a content sub-entity (521) for audio content (424) and a metadata sub-entity (522) for associated metadata. [EEE7] a metadata frame for a corresponding channel frame including a plurality of different variants of metadata (434, 444); A representation (401) indicating which transformation (434) of metadata should be used to render the corresponding channel frame, The method (600) of any one of EEE1 to EEE6. [EEE8] determining a speaker channel bed (421) indicative of audio content of one or more audio signals of the set of audio signals, the speaker channel bed (421) comprising a sequence of speaker channel frames; the first audio program frame of the object-based audio program comprising a first speaker channel frame of the speaker channel bed (421); inserting the first speaker channel frame into a speaker channel sub-stream entity (520) of the plurality of sub-stream entities (520) of the first container (501), The method (600) of any one of EEE1 to EEE7. [EEE9] The method (600) according to EEE8, wherein the presentation (401) also includes the speaker channel sub-stream entities (520). [EEE10] The method (600) of any one of EEE8 or 9, wherein the bed of speaker channels (421) comprises one or more speaker channels to be presented respectively by one or more speakers of a presentation environment. [EEE11] The method (600) further includes providing speaker-related metadata (431, 441) for the speaker channel bed (421); the speaker related metadata (431, 441) comprises a sequence of speaker related metadata frames; a first speaker related metadata frame from the sequence of speaker related metadata frames is inserted into the speaker channel sub-stream entity (520); The method (600) of any one of EEE8 to 10. [EEE12] The method (600) of any one of claims 8 to 11, wherein the bed of speaker channels (421) includes one or more of 2.0 channels, 5.1 channels and / or 7.1 channels. [EEE13] The method (600) of any one of EEE1-12, wherein the collection of object channels (424) comprises a plurality of object channels (424). [EEE14] The method (600) of any one of EEE1-13, wherein the object channel (424) is presented by a combination of speakers (36) in a presentation environment. [EEE15] The method (600) of EEE14, wherein the object relational metadata (434, 444) of an object channel (424) indicates a location within the presentation environment from which the object channel (424) should be rendered. [EEE16] the position of the object channel (424) varies over time; the combination of speakers (36) for rendering the object channel (424) varies along the sequence of object channel frames of the object channel (424); and / or the panning of the speakers (36) of the combination of speakers (36) varies along a sequence of the object channel frames of the object channel (424); The method (600) described in EEE14 or 15. [EEE17] The method (600) of any one of EEE1 to EEE16, wherein the bitstream (500) is an AC-4 bitstream. [EEE18] The method (600) of any one of EEE1 to EEE17, wherein the collection of audio signals is indicative of captured audio content. [EEE19] The presentation (401) includes target device configuration data for a target device configuration; the target device configuration data indicating a set of sub-stream entities (520) and / or metadata transformations (434) from the plurality of sub-stream entities (520) to be used to render the presentation (401) on the target device configuration; The method (600) of any one of EEE1 to EEE18. [EEE20] one, two or more of the plurality of sub-streams form a sub-stream group; A display (401) showing the sub-stream group; The method (600) of any one of EEE1 to EEE19. [EEE21] The method (600) as described in EEE20, further comprising the step of inserting data for identifying said one, two or more of said plurality of sub-streams into a sub-stream group entity of said presentation section (510), said sub-stream group entity including data for defining said sub-stream group. [EEE22] A bitstream (500) representing an object-based audio program, comprising: said bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of said object-based audio program; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames and a corresponding first set of object related metadata frames; the first set of object channel frames indicative of audio content of at least some audio signals of a set of audio signals; the first container (501) containing a number of substream entities (520) for a number of substreams (411, 412, 413, 414) of the object-based audio program; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510) having presentation data; the presentation data indicative of at least one presentation (401) of the object-based audio program; the presentation (401) includes a collection of sub-stream entities (520) from said plurality of sub-stream entities (520) to be presented simultaneously, Bitstream. [EEE23] the first audio program frame includes a first speaker channel frame of a bed of speaker channels (421); a bed of speaker channels (421) indicating the audio content of one or more audio signals of the collection of audio signals; the plurality of sub-stream entities (520) includes a speaker channel sub-stream entity (520) for the first speaker channel frame; A bitstream as described in EEE22. [EEE24] A method for generating a personalized audio program from a bitstream (500) containing an object-based audio program, comprising: the bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames of a set of object channels (424) and a corresponding first set of object related metadata frames; the set of object channels (424) representing audio content of at least some audio signals of the set of audio signals; the first container (501) containing a number of substream entities (520) for a number of substreams (411, 412, 413, 414) of the object-based audio program; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510); The method comprises: extracting presentation data from the presentation section (510), the presentation data indicating a presentation (401) for the personalized audio program, the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously; extracting one or more object channel frames and corresponding one or more object relation metadata frames from the set of object channel sub-stream entities (520) of the first container (501) based on the presentation data, method. [EEE25] the first audio program frame includes a first speaker channel frame of a bed of speaker channels (421); a bed of speaker channels (421) indicating the audio content of one or more audio signals of the collection of audio signals; the plurality of sub-stream entities (520) includes a speaker channel sub-stream entity (520) for the first speaker channel frame; the method further comprising extracting the first speaker channel frame from the speaker channel sub-stream entity (520) of the first container (501) based on the presentation data, The method described in EEE24. [EEE26] A system (3) for generating a bitstream (500) representing an object-based audio program, the bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) in the sequence of containers (501) comprising a plurality of substream entities (520) for a plurality of substreams (411, 412, 413, 414) of the object-based audio program; the first container (501) further comprising a presentation section (510); the system (3) further comprising: determining a set of object channels (424) indicative of audio content of at least some of the set of audio signals, the set of object channels (424) comprising a sequence of a set of object channel frames; determining a set of object related metadata (434, 444) for the set of object channels (424), the set of object related metadata (434, 444) comprising a sequence of a set of object related metadata frames; a first audio program frame of the object based audio program comprising a first set of object channel frames and a corresponding first set of object related metadata frames; inserting the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel sub-stream entities (520) of the plurality of sub-stream entities (520) of the first container (501); inserting presentation data into said presentation section (510), said presentation data representing at least one presentation (401); said at least one presentation (401) including a collection of sub-stream entities (520) from said plurality of sub-stream entities (520) to be presented simultaneously; system. [EEE27] A system (7) for generating a personalized audio program from a bitstream (500) containing an object-based audio program, comprising: the bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames of a set of object channels (424) and a corresponding first set of object related metadata frames; the set of object channels (424) representing audio content of at least some audio signals of the set of audio signals; the first container (501) containing a number of substream entities (520) for a number of substreams (411, 412, 413, 414) of the object-based audio program; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510); The system (7) is extracting presentation data from the presentation section (510), the presentation data indicating a presentation (401) for the personalized audio program, the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously; extracting one or more object channel frames and corresponding one or more object relation metadata frames from the set of object channel sub-stream entities (520) of the first container (501) based on the presentation data; system.

[0086] Several aspects will be described. [Aspect 1] A method (600) for generating a bitstream (500) representing an object-based audio program, the object-based audio program comprising a plurality of substreams; the bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) of the sequence of containers (501) comprising a plurality of substream entities (520) for each of the plurality of substreams (411, 412, 413, 414); a substream entity comprising data relating to frames of a corresponding substream; the first container (501) further comprising a presentation section (510); the method (600) comprising: determining (601) a set of object channels (424) indicative of audio content of at least some of the set of audio signals, the set of object channels (424) comprising a sequence of a set of object channel frames; providing (602) a set of object relational metadata (434, 444) for the set of object channels (424), the set of object relational metadata (434, 444) comprising a sequence of sets of object relational metadata frames; a first audio program frame of the object based audio program comprising a first set of object channel frames and a corresponding first set of object relational metadata frames, an object channel to be presented by a combination of speakers of a presentation environment, the object relational metadata for an object channel indicating a location within the presentation environment from which the object channel should be rendered; inserting (603) the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel sub-stream entities (520) of the plurality of sub-stream entities (520) of the first container (501); inserting (604) presentation data into the presentation section (510), the presentation data representing at least one presentation (401); the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously, method. [Aspect 2] the presentation data is segmented into different presentation data entities (512) for different presentations (401, 402, 403); The method further includes inserting table of contents data, referred to as TOC data, into the presentation section (510), the TOC data comprising: the location of the different presentation data entities (512) within the presentation section (510); and / or indicating identifiers for the different presentation data entities (512) contained within the presentation section (510), The method (600) of embodiment 1. [Aspect 3] a metadata frame for a corresponding channel frame including a plurality of different variants of metadata (434, 444); A representation (401) indicating which transformation (434) of metadata should be used to render the corresponding channel frame, The method (600) of embodiment 1 or 2. Aspect 4 determining a speaker channel bed (421) indicative of audio content of one or more audio signals of the set of audio signals, the speaker channel bed (421) comprising a sequence of speaker channel frames; the first audio program frame of the object-based audio program comprising a first speaker channel frame of the speaker channel bed (421); inserting the first speaker channel frame into a speaker channel sub-stream entity (520) of the plurality of sub-stream entities (520) of the first container (501), The method (600) of any one of embodiments 1 to 3. Aspect 5 The method (600) of embodiment 4, wherein the speaker channel bed (421) includes one or more speaker channels to be presented by one or more speakers of a presentation environment, respectively. Aspect 6 The method (600) further includes providing speaker-related metadata (431, 441) for the speaker channel bed (421); the speaker related metadata (431, 441) comprises a sequence of speaker related metadata frames; a first speaker related metadata frame from the sequence of speaker related metadata frames is inserted into the speaker channel sub-stream entity (520); The method (600) of embodiment 4 or 5. Aspect 7 the position of the object channel (424) varies over time; the combination of speakers (36) for rendering the object channel (424) varies along the sequence of object channel frames of the object channel (424); and / or the panning of the speakers (36) of the combination of speakers (36) varies along a sequence of the object channel frames of the object channel (424); The method (600) of any one of embodiments 1 to 6. Aspect 8 The presentation (401) includes target device configuration data for a target device configuration; the target device configuration data indicating a set of sub-stream entities (520) and / or metadata transformations (434) from the plurality of sub-stream entities (520) to be used to render the presentation (401) on the target device configuration; The method (600) of any one of embodiments 1 to 7. Aspect 9 one, two or more of the plurality of sub-streams form a sub-stream group; A display (401) showing the sub-stream group; the method further comprising inserting data identifying said one, two or more of said plurality of substreams into a substream group entity of said presentation section (510), said substream group entity including data defining said substream group; The method (600) of any one of embodiments 1 to 8. Aspect 10 A bitstream (500) representing an object-based audio program, comprising: the bitstream (500) comprises a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program, the object-based audio program comprising a plurality of substreams; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames and a corresponding first set of object related metadata frames; an object channel frame is to be presented by a combination of speakers in a presentation environment, and said object relations metadata frame of an object channel frame indicates a location within said presentation environment from which that object channel frame should be rendered; the first set of object channel frames indicative of audio content of at least some audio signals of a set of audio signals; the first container (501) includes a plurality of sub-stream entities (520) for each of the plurality of sub-streams (411, 412, 413, 414); a sub-stream entity includes data relating to frames of a corresponding sub-stream; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510) having presentation data; the presentation data indicative of at least one presentation (401) of the object-based audio program; the presentation (401) includes a collection of sub-stream entities (520) from said plurality of sub-stream entities (520) to be presented simultaneously, Bitstream. Aspect 11 the first audio program frame includes a first speaker channel frame of a bed of speaker channels (421); a bed of speaker channels (421) indicating the audio content of one or more audio signals of the collection of audio signals; the plurality of sub-stream entities (520) includes a speaker channel sub-stream entity (520) for the first speaker channel frame; 11. The bitstream according to embodiment 10. Aspect 12 A method for generating a personalized audio program from a bitstream (500) containing an object-based audio program, comprising: the bitstream (500) comprises a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program, the object-based audio program comprising a plurality of substreams; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames of a set of object channels (424) and a corresponding first set of object related metadata frames; an object channel frame is to be presented by a combination of speakers in a presentation environment, and said object relations metadata frame of an object channel frame indicates a location within said presentation environment from which that object channel frame should be rendered; the set of object channels (424) representing audio content of at least some audio signals of the set of audio signals; the first container (501) includes a plurality of sub-stream entities (520) for each of the plurality of sub-streams (411, 412, 413, 414); a sub-stream entity includes data relating to frames of a corresponding sub-stream; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510); The method comprises: extracting presentation data from the presentation section (510), the presentation data indicating a presentation (401) for the personalized audio program, the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously; extracting one or more object channel frames and corresponding one or more object relation metadata frames from the set of object channel sub-stream entities (520) of the first container (501) based on the presentation data, method. Aspect 13 the first audio program frame includes a first speaker channel frame of a bed of speaker channels (421); a bed of speaker channels (421) indicating the audio content of one or more audio signals of the collection of audio signals; the plurality of sub-stream entities (520) includes a speaker channel sub-stream entity (520) for the first speaker channel frame; the method further comprising extracting the first speaker channel frame from the speaker channel sub-stream entity (520) of the first container (501) based on the presentation data, The method of embodiment 12. Aspect 14 1. A system (3) for generating a bitstream (500) representing an object-based audio program, said bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of said object-based audio program; said object-based audio program comprising a plurality of substreams; a first container (501) of said sequence of containers (501) comprising a plurality of substream entities (520) for each of said plurality of substreams (411, 412, 413, 414); a substream entity comprising data relating to frames of a corresponding substream; said first container (501) further comprising a presentation section (510); determining a set of object channels (424) indicative of audio content of at least some of the set of audio signals, the set of object channels (424) comprising a sequence of a set of object channel frames; determining a set of object relational metadata (434, 444) for a set of object channels (424), the set of object relational metadata (434, 444) comprising a sequence of sets of object relational metadata frames; a first audio program frame of the object based audio program comprising a first set of object channel frames and a corresponding first set of object relational metadata frames, an object channel being presented by a combination of speakers of a presentation environment, the object relational metadata for an object channel indicating a location within the presentation environment from which the object channel should be rendered; inserting the first set of object channel frames and the first set of object relation metadata frames into a respective set of object channel sub-stream entities (520) of the plurality of sub-stream entities (520) of the first container (501); inserting presentation data into said presentation section (510), said presentation data representing at least one presentation (401); said at least one presentation (401) including a collection of sub-stream entities (520) from said plurality of sub-stream entities (520) to be presented simultaneously; system. Aspect 15 1. A system (7) for generating a personalized audio program from a bitstream (500) including an object-based audio program, the object-based audio program including a plurality of substreams; the bitstream (500) comprising a sequence of containers (501) for a corresponding sequence of audio program frames of the object-based audio program; a first container (501) of the sequence of containers (501) containing a first audio program frame of the object-based audio program; the first audio program frames including a first set of object channel frames of a set of object channels (424) and a corresponding first set of object related metadata frames; an object channel frame is to be presented by a combination of speakers in a presentation environment, and said object relations metadata frame of an object channel frame indicates a location within said presentation environment from which that object channel frame should be rendered; the set of object channels (424) representing audio content of at least some audio signals of the set of audio signals; the first container (501) includes a plurality of sub-stream entities (520) for each of the plurality of sub-streams (411, 412, 413, 414); a sub-stream entity includes data relating to frames of a corresponding sub-stream; the plurality of sub-stream entities (520) each including a set of object channel sub-stream entities (520) for the first set of object channel frames; the first container (501) further comprises a presentation section (510); The system (7) is extracting presentation data from the presentation section (510), the presentation data indicating a presentation (401) for the personalized audio program, the presentation (401) including a collection of sub-stream entities (520) from the plurality of sub-stream entities (520) to be presented simultaneously; extracting one or more object channel frames and corresponding one or more object relation metadata frames from the set of object channel sub-stream entities (520) of the first container (501) based on the presentation data; system.

Claims

1. 1. A method for decoding an audio program from an encoded bitstream, the encoded bitstream including a sequence of containers for a corresponding sequence of audio program frames, each container including, for an audio program frame, presentation data and a plurality of substream entities, each substream entity including object channel audio data and metadata, the method comprising: receiving the encoded bitstream; extracting the presentation data from the container for an audio program frame of the encoded bitstream, the presentation data indicating a presentation of the audio program, the presentation data being used to identify one or more sub-stream entities from the plurality of sub-stream entities, the one or more sub-stream entities being used for the audio program to be rendered; and decoding metadata and object channel audio data corresponding to each of the one or more sub-stream entities; the metadata indicating a location within a presentation environment from which corresponding object channel audio data should be rendered; method.

2. 1. A system for decoding an audio program from an encoded bitstream, the encoded bitstream including a sequence of containers for a corresponding sequence of audio program frames, each container including, for an audio program frame, presentation data and a plurality of substream entities, each substream entity including object channel audio data and metadata, the system comprising: a receiver for receiving the encoded bitstream; an extractor for extracting the presentation data from the container for an audio program frame of the encoded bitstream, the presentation data indicating a presentation of the audio program, the presentation data being used to identify one or more sub-stream entities from the plurality of sub-stream entities, the one or more sub-stream entities being used for the audio program to be rendered; a decoder for decoding metadata and object channel audio data corresponding to each of the one or more sub-stream entities; the metadata indicating a location within a presentation environment from which corresponding object channel audio data should be rendered; system.

3. A non-transitory computer readable storage medium having stored thereon a computer program for causing a computer to perform the method of claim 1.

Images