Layered coding for compressing sound or soundfield representations

Abstract

The present disclosure relates to layered coding for compressing a sound or soundfield representation. The compressed sound representation comprises a base compressed sound representation comprising a plurality of components, base side information for decoding the base compressed sound representation into a base reconstructed sound representation of the sound or soundfield, and enhancement side information comprising parameters for improving the base reconstructed sound representation. The layered coding method comprises subdividing the plurality of components into a plurality of component groups and assigning each group to a respective one of a plurality of hierarchical layers comprising a base layer and one or more hierarchical enhancement layers, adding the base side information to the base layer, and determining from the enhancement side information a plurality of portions of the enhancement side information and assigning each of the plurality of portions of the enhancement side information to a respective hierarchical layer of the plurality of layers, wherein each portion of the enhancement side information comprises parameters for improving the reconstructed sound representation obtainable from data comprised in the respective layer and any layers below the respective layer.

Description

[0001] This application is a divisional application of the invention patent application with application number 201680058435.9, application date October 7, 2016, and invention title "Layered encoding and decoding for compressing sound or sound field representation".

[0002] Cross-reference to related applications

[0003] This application claims priority to European Patent Application No. 15306589.1, filed October 8, 2015, and European Patent Application No. 15306653.5, filed October 15, 2015, as well as U.S. Patent Application Nos. 62 / 361461 and 62 / 361416, which are incorporated herein by reference in their entirety. Technical Field

[0004] This document relates to methods and apparatus for layered audio encoding and decoding. In particular, it relates to methods and apparatus for layered audio encoding and decoding for compressing sound (or sound field) representations, such as high-order high-fidelity stereo reproduction (HOA) sound (or sound field) representations. Background Technology

[0005] For streaming sound (or sound field) on a transmission channel under time-varying conditions, layered encoding and decoding is a method that adapts the quality of the received sound representation to the transmission conditions and is particularly suitable for avoiding unwanted signal loss.

[0006] For layered encoding and decoding, the sound (or sound field) representation is often subdivided into a relatively small, high-priority base layer and additional enhancement layers of decreasing priority and arbitrary size. Typically, each enhancement layer is assumed to contain incremental information to supplement the information of all lower layers, thereby improving the quality of the sound (or sound field) representation. The amount of error protection used for the transmission of each layer is controlled based on their priority. In particular, the base layer has high error protection, which is reasonable and cost-effective due to its small size.

[0007] However, there is still a need for layered codec schemes for compressed representations (extended versions) of special types of sounds or sound fields, such as compressed HOA sounds or sound field representations.

[0008] This document addresses the aforementioned issues. It specifically describes a layered encoding / decoding method and encoder / decoder for compressing sound and representing sound fields. Summary of the Invention

[0009] According to one aspect, a hierarchical coding method for a compressed sound representation of a sound or sound field is described. The compressed sound representation may include a basic compressed sound representation comprising multiple components. These multiple components may be supplementary components. The compressed sound representation may further include basic auxiliary information for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The compressed sound representation may further include enhancement auxiliary information, which includes parameters for improving (e.g., enhancing) the basic reconstructed sound representation. The method may include subdividing (e.g., grouping) the multiple components into multiple component groups. The method may further include assigning (e.g., adding) each of the multiple groups to a corresponding hierarchical layer among multiple hierarchical layers. This assignment may indicate the correspondence between the groups and layers. Components assigned to a corresponding layer may be referred to as being included in that layer. The number of groups may correspond to (e.g., equal to) the number of layers. The multiple layers may include a base layer and one or more hierarchical enhancement layers. The multiple hierarchical layers may be ordered from the base layer, via a first enhancement layer, a second enhancement layer, etc., up to the overall highest enhancement layer (overall highest layer). The method may further include adding basic auxiliary information to a base layer (e.g., including the basic auxiliary information in the base layer or assigning the basic auxiliary information to the base layer for transmission or storage purposes). The method may further include determining multiple portions of the enhanced auxiliary information from the enhanced auxiliary information. The method may further include allocating (e.g., adding) each of the multiple portions of the enhanced auxiliary information to a corresponding one of the multiple layers. Each portion of the enhanced auxiliary information may include parameters for improving the reconstructed (e.g., decompressed) sound representation obtainable from data included (e.g., allocated or added to) the corresponding layer and any layers below that corresponding layer. Layered encoding may be used for transmission over a transmission channel or for storage on a suitable storage medium (e.g., CD, DVD, Blu-ray disc). TM It is implemented for the purpose of ).

[0010] As configured above, the proposed method enables efficient application of layered encoding and decoding to compressed audio representations containing multiple components and basic and enhanced auxiliary information (e.g., independent basic and enhanced auxiliary information) with the properties described above. The proposed method specifically ensures that each layer includes appropriate auxiliary information for reconstructing the audio representation from components included in any layer up to the layer of interest. Here, "layer up to the layer of interest" is understood to include, for example, the basic layer, the first enhancement layer, the second enhancement layer, etc., up to the layer of interest. Therefore, regardless of the actual highest usable layer (e.g., a layer below the lowest layer that is not effectively received, such that all layers below the highest usable layer and the highest usable layer itself have been effectively received), even if the reconstructed audio representation may differ from the complete (e.g., full) audio representation, the decoder will still be able to improve or enhance the reconstructed audio representation. Specifically, regardless of the actual highest usable layer, it is sufficient for the decoder to decode the payload of enhanced auxiliary information for only a single layer (i.e., for the highest usable layer) to improve or enhance the reconstructed audio representation based on all components included in the layers up to the actual highest usable layer. That is, for each time interval (e.g., frame), only a single payload of enhanced auxiliary information needs to be decoded. On the other hand, the proposed method allows full utilization of the reduced bandwidth required when applying layered encoding and decoding.

[0011] In an embodiment, the components of the basic compressed sound representation may correspond to a mono signal (e.g., a transmit signal or a mono transmit signal). This mono signal may represent a sequence of coefficients or the primary sound signal represented by the HOA. The mono signal may be quantized.

[0012] In embodiments, the basic auxiliary information may include information that independently specifies the decoding (e.g., decompression) of one or more of the plurality of components, independent of other components. For example, the basic auxiliary information may represent auxiliary information related to an individual mono signal and independent of other mono signals. Therefore, the basic auxiliary information may be referred to as independent basic auxiliary information.

[0013] In an embodiment, the enhancement auxiliary information may represent enhancement auxiliary information. The enhancement auxiliary information may include prediction parameters for improving (e.g., enhancing) the basic compressed sound representation that can be obtained from the basic compressed sound representation and the basic auxiliary information.

[0014] In embodiments, the method may further include generating a transport stream for the transmission of data for multiple layers (e.g., data allocated or added to or additionally included in each layer). The base layer may have the highest transmission priority, and the hierarchical enhancement layers may have decreasing transmission priorities. That is, the transmission priority may decrease from the base layer to the first enhancement layer, from the first enhancement layer to the second enhancement layer, and so on. The amount of error protection for the transmission of data for these multiple layers may be controlled according to the corresponding transmission priority. Therefore, reliable transmission of at least some lower layers can be ensured, while on the other hand, the overall required bandwidth is reduced by not applying excessive error protection to higher layers.

[0015] In an embodiment, the method may further include generating a transport layer packet containing data for each of the plurality of layers. For example, for each time interval (e.g., a frame), a corresponding transport layer packet may be generated for each of the plurality of layers.

[0016] In an embodiment, the compressed audio representation may further include additional basic auxiliary information for decoding the basic compressed audio representation into a basic reconstructed audio representation. The additional basic auxiliary information may include information specifying the decoding of one or more of a plurality of components that depends on the decoding of the other components. The method may further include decomposing the additional basic auxiliary information into multiple portions of the additional basic auxiliary information. The method may further include adding portions of the additional basic auxiliary information to a base layer (e.g., for transmission or storage purposes, such as including portions of the additional basic auxiliary information in the base layer or assigning portions of the additional basic auxiliary information to a base layer). Each portion of the additional basic auxiliary information may correspond to a corresponding layer and may include information specifying that the decoding of one or more components assigned to that corresponding layer depends (only) on the decoding of other components assigned to that corresponding layer and any layers below that corresponding layer. That is, each portion of the additional basic auxiliary information specifies the component in that corresponding layer to which that portion of the additional basic auxiliary information corresponds, without referencing any other components assigned to layers higher than that corresponding layer.

[0017] In this configuration, the proposed method avoids fragmentation of supplementary basic auxiliary information by adding all parts to the base layer. In other words, all parts of the supplementary basic auxiliary information are included in the base layer. The decomposition of the supplementary basic auxiliary information ensures that for each layer, a portion of the supplementary basic auxiliary information is available without needing to know the components in higher layers. Therefore, regardless of the highest usable layer, the decoder only needs to decode the supplementary basic auxiliary information included up to the highest usable layer.

[0018] In embodiments, the additional basic auxiliary information may include information specifying one or more of the plurality of components that are dependent on the decoding (e.g., decompression) of the other components. For example, the additional basic auxiliary information may represent auxiliary information related to an individual mono signal that is dependent on other mono signals. Therefore, the additional basic auxiliary information may be referred to as dependent basic auxiliary information.

[0019] In an embodiment, the compressed audio representation can be processed over consecutive time intervals, such as time intervals of equal size. A consecutive time interval can be a frame. Therefore, the method can operate on a frame-by-frame basis; that is, the compressed audio representation can be encoded in a frame-by-frame manner. The compressed audio representation can be available for each consecutive time interval (e.g., for each frame). That is, the compression operation to obtain the compressed audio representation can be operated on a frame-by-frame basis.

[0020] In an embodiment, the method may further include generating configuration information that indicates the components of the basic compressed audio representation assigned to each layer. Therefore, the decoder can quickly obtain the information needed for decoding without unnecessarily parsing the received data payload.

[0021] According to another aspect, a hierarchical coding method for a compressed sound representation of a sound or sound field is described. The compressed sound representation may include a basic compressed sound representation comprising multiple components. These multiple components may be supplementary components. The compressed sound representation may further include basic auxiliary information (e.g., independent basic auxiliary information) and third information (e.g., dependent basic auxiliary information) for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The basic auxiliary information may include information that independently specifies the decoding of one or more of the multiple components, independent of other components. Additional basic auxiliary information may include information that specifies the decoding of one or more of the multiple components dependent on the other components. The method may include subdividing (e.g., grouping) the multiple components into multiple component groups. The method may further include assigning (e.g., adding) each of the multiple groups to a corresponding hierarchical layer in a plurality of hierarchical layers. This assignment may indicate the correspondence between each group and layer. The component assigned to the corresponding layer may be referred to as being included in that layer. The number of groups may correspond to (e.g., equal to) the number of layers. The multiple layers may include a base layer and one or more hierarchical enhancement layers. The method may further include adding basic auxiliary information to a base layer (e.g., including the basic auxiliary information in the base layer or assigning the basic auxiliary information to the base layer for transmission or storage purposes). The method may further include decomposing the additional basic auxiliary information into multiple parts of the additional basic auxiliary information and adding the parts of the additional basic auxiliary information to the base layer (e.g., including the parts of the additional basic auxiliary information in the base layer or assigning the parts of the additional basic auxiliary information to the base layer for transmission or storage purposes). Each part of the additional basic auxiliary information may correspond to a corresponding layer and include information specifying the decoding of one or more components assigned to that corresponding layer and any other components assigned to that corresponding layer and any layers below that corresponding layer.

[0022] In this configuration, the proposed method ensures that, for each layer, appropriate additional basic auxiliary information is available for decoding components included in any layer up to that corresponding layer, without needing to effectively receive or decode (or in general, know) any higher layers. In the case of compressed HOA representations, the proposed method ensures that, in the vector coding mode, appropriate V vectors are available for all components belonging to layers up to the highest usable layer. Specifically, the proposed method excludes cases where elements of the V vectors corresponding to components in higher layers are not explicitly signaled. Therefore, the information included in layers up to the highest usable layer is sufficient for decoding (e.g., decompressing) any component belonging to layers up to the highest usable layer. Thus, proper decompression of the corresponding reconstructed HOA representation of lower layers is ensured even if higher layers are not effectively received by the decoder. On the other hand, the proposed method allows full utilization of the reduced bandwidth required when applying layered encoding and decoding.

[0023] Embodiments in this respect may be related to embodiments described above.

[0024] According to another aspect, a hierarchical coding method for a compressed sound representation of a sound or sound field is described. The compressed sound representation may be encoded in multiple hierarchical layers. These hierarchical layers may include a base layer and one or more hierarchical enhancement layers. The multiple layers may be assigned components of the basic compressed sound representation of the sound or sound field. In other words, the multiple layers may include components of basic compression auxiliary information. These components may be assigned to each layer in a component group. The multiple components may be supplementary components. The base layer may include basic auxiliary information for decoding the basic compressed sound representation. Each layer may contain a portion of enhancement auxiliary information including parameters for improving the basic reconstructed sound representation, which may be obtained from data included in the corresponding layer and any layer below that corresponding layer. The method may include receiving data payloads respectively corresponding to the multiple hierarchical layers. The method may further include determining a first layer index indicating the highest usable layer among the multiple layers to be used for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The method may further include using the basic auxiliary information to obtain the basic reconstructed sound representation from the components assigned to the highest usable layer and any layer below that highest usable layer. The method may further include determining a second-level index that indicates which portion of the enhancement auxiliary information should be used to improve (e.g., enhance) the basic reconstructed sound representation. The method may include reconstructing a sound representation of a sound or sound field from the basic reconstructed sound representation by referring to the second-level index.

[0025] In such a configuration, the proposed method ensures that the reconstructed sound representation has the best quality by using available (e.g., effectively received) information to the best possible extent.

[0026] In an embodiment, the components of the basic compressed sound representation may correspond to a mono signal (e.g., a mono transmission signal). This mono signal may represent a sequence of coefficients or the main sound signal represented by the HOA. The mono signal may be quantized.

[0027] In embodiments, the basic auxiliary information may include information that independently specifies the decoding (e.g., decompression) of one or more of the plurality of components, independent of other components. For example, the basic auxiliary information may represent auxiliary information related to an individual mono signal and independent of other mono signals. Therefore, the basic auxiliary information may be referred to as independent basic auxiliary information.

[0028] In an embodiment, the enhancement auxiliary information may represent enhancement auxiliary information. The enhancement auxiliary information may include prediction parameters for improving (e.g., enhancing) the basic compressed sound representation that can be obtained from the basic compressed sound representation and the basic auxiliary information.

[0029] In an embodiment, the method may further include, for each layer, determining whether the corresponding layer has been effectively received. The method may further include determining the first layer index as the layer index of the layer immediately following the lowest layer that was not effectively received.

[0030] In an embodiment, determining the second-level index may involve determining that the second-level index is equal to the first-level index, or determining the index value that indicates no enhancement auxiliary information is used when obtaining the reconstructed sound representation as the second-level index. In the latter case, the reconstructed sound representation may be equal to the basic reconstructed sound representation.

[0031] In an embodiment, the data payload may be received and processed over consecutive time intervals (e.g., time intervals of equal size). The consecutive time intervals may be frames. Therefore, the method can operate on a frame-by-frame basis. The method may further include determining that a second-level index equals a first-level index if the compressed audio representations of the consecutive time intervals can be decoded independently of each other.

[0032] In an embodiment, the data payload may be received and processed for consecutive time intervals (e.g., time intervals of equal size). A consecutive time interval may be a frame. Therefore, the method can operate on a frame-by-frame basis. The method may further include, for a given time interval within the consecutive time interval, determining whether a corresponding layer has been effectively received if the compressed audio representations of the consecutive time intervals cannot be decoded independently of each other. The method may further include determining the first layer index of a given time interval as the smaller of the first layer index of the time interval preceding the given time interval and the layer index of the layer immediately below the lowest layer that was not effectively received.

[0033] In an embodiment, the method may further include, for a given time interval, if the compressed audio representations of consecutive time intervals cannot be decoded independently of each other, determining whether the first-level index of the given time interval is equal to the first-level index of a prior time interval. The method may further include, if the first-level index of the given time interval is equal to the first-level index of a prior time interval, determining that the second-level index of the given time interval is equal to the first-level index of the given time interval. The method may further include, if the first-level index of the given time interval is not equal to the first-level index of a prior time interval, determining that the index value indicating that no enhancement auxiliary information is used when obtaining the reconstructed audio representation is the second-level index.

[0034] In an embodiment, the base layer may include at least a portion of additional basic auxiliary information corresponding to the corresponding layer and including information specifying the decoding of one or more components assigned to the corresponding layer dependent on other components assigned to the corresponding layer and any layers below the corresponding layer. The method may further include decoding each portion of the additional basic auxiliary information by referencing components assigned to its corresponding layer and any layers below the corresponding layer. The method may further include correcting the portion of the additional basic auxiliary information by referencing components assigned to the highest usable layer and any layers between the highest usable layer and the corresponding layer. Using the basic auxiliary information and the corrected portion of the additional basic auxiliary information obtained from the portion of the additional basic auxiliary information corresponding to the layers up to the highest usable layer, a basic reconstructed sound representation can be obtained from the components assigned to the highest usable layer and any layers below the highest usable layer.

[0035] In embodiments, the additional basic auxiliary information may include information that specifies one or more of a plurality of components and is dependent on the decoding (e.g., decompression) of other components. For example, the additional basic auxiliary information may represent auxiliary information related to an individual mono signal and dependent on other mono signals. Therefore, the additional basic auxiliary information may be referred to as dependent basic auxiliary information.

[0036] According to another aspect, a decoding method for a compressed sound representation of a sound or sound field is described. The compressed sound representation may be encoded in multiple hierarchical layers. The multiple hierarchical layers may include a base layer and one or more hierarchical enhancement layers. The multiple layers may be assigned components of the basic compressed sound representation of the sound or sound field. In other words, the multiple layers may include components of basic compression auxiliary information. These components may be assigned to each layer in a component group. The multiple components may be supplementary components. The base layer may include basic auxiliary information for decoding the basic compressed sound representation. The base layer may further include at least a portion of additional basic auxiliary information corresponding to the corresponding layer and including information specifying the decoding of one or more components assigned to the corresponding layer dependent on other components assigned to the corresponding layer and any layers below the corresponding layer. The method may include receiving data payloads respectively corresponding to the multiple hierarchical layers. The method may further include determining a first layer index indicating the highest usable layer among the multiple layers to be used for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The method may further include decoding, for each portion of the additional basic auxiliary information, by referring to the components assigned to its corresponding layer and any layers below the corresponding layer. The method may further include correcting, for each portion of the additional basic auxiliary information, a portion of the additional basic auxiliary information by referring to components assigned to the highest usable layer and any layer between the highest usable layer and the corresponding layer. By using the basic auxiliary information and the corrected portions of the additional basic auxiliary information obtained from portions corresponding to layers up to the highest usable layer, a basic reconstructed sound representation can be obtained from components assigned to the highest usable layer and any layer below the highest usable layer. The method may further include determining a second layer index, which is equal to the first layer index or indicates that enhanced auxiliary information is omitted during decoding.

[0037] With this configuration, the proposed method ensures that the additional basic auxiliary information used to decode the basic compressed audio representation does not contain redundant elements, thereby making the actual decoding of the basic compressed audio representation more efficient.

[0038] Embodiments in this respect may be related to embodiments in the foregoing respect.

[0039] According to another aspect, an encoder for layered encoding of a compressed sound representation for a sound or sound field is described. The compressed sound representation may include a basic compressed sound representation comprising multiple components. These multiple components may be supplementary components. The compressed sound representation may further include basic auxiliary information for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The compressed sound representation may further include enhancement auxiliary information, which includes parameters for improving (e.g., enhancing) the basic reconstructed sound representation. The encoder may include some or all of the method steps configured to implement the methods according to the first and second aspects mentioned above.

[0040] According to another aspect, a decoder for decoding a compressed sound representation of a sound or sound field is described. The compressed sound representation may be encoded in multiple hierarchical layers. These hierarchical layers may include a base layer and one or more hierarchical enhancement layers. The multiple layers may be assigned components of the basic compressed sound representation of the sound or sound field. In other words, the multiple layers may include components of the basic compression auxiliary information. These components may be assigned to corresponding layers in respective component groups. The multiple components may be supplementary components. The base layer may include basic auxiliary information for decoding the basic compressed sound representation. Each layer may include portions of enhancement auxiliary information including parameters for improving (e.g., enhancing) the basic reconstructed sound representation, which may be obtained from data included in the corresponding layer and any layers below that corresponding layer. The decoder may include a processor configured to implement some or all of the method steps according to the third and fourth aspects mentioned above.

[0041] According to other aspects, methods, apparatus, and systems relate to decoding a compressed high-order high-fidelity stereo reproduction (HOA) sound representation of a sound or sound field. The apparatus may have a receiver configured or the method configured to receive a bitstream comprising a compressed HOA representation corresponding to a plurality of hierarchical layers including a base layer and one or more hierarchical enhancement layers. The plurality of layers are assigned components of the basic compressed sound representation of the sound or sound field, with the components allocated to each layer in a component group. The apparatus may have a decoder configured or the method configured to decode the compressed HOA representation based on basic auxiliary information associated with the base layer and enhancement auxiliary information associated with the one or more hierarchical enhancement layers. The basic auxiliary information may include basic independent auxiliary information relating to a first volume mono signal, which will be decoded independently of other mono signals. Each of the one or more hierarchical enhancement layers may include a portion of enhancement auxiliary information containing parameters for improving the basic reconstructed sound representation, which may be derived from data included in the corresponding layer and any layers below the corresponding layer.

[0042] The basic independent auxiliary information may indicate that the first volume mono signal represents a directional signal with an incident direction. This basic auxiliary information may further include basic dependency auxiliary information relating to a second volume mono signal, which will depend on the decoding of other mono signals. This basic dependency auxiliary information may include a vector-based signal directionally distributed within the sound field, wherein the directional distribution is specified by a vector. The components of this vector are set to zero and are not part of the compressed vector representation.

[0043] The components of the basic compressed sound representation may correspond to a coefficient sequence representing the HOA representation or a mono signal representing the primary sound signal. The bitstream includes data payloads corresponding to multiple hierarchical layers. The enhancement auxiliary information may include parameters related to at least one of the following: spatial prediction, sub-band directional signal synthesis, and parameter environment replication. The enhancement auxiliary information may include information that allows prediction of missing portions of the sound or sound field from the directional signals. For each layer, it may be further determined whether the corresponding layer has been effectively received and the layer index of the layer immediately following the lowest layer that has not been effectively received.

[0044] According to another aspect, a software program is described. This software program is adapted to execute on a processor, and when implemented on a computing device, it is suitable for carrying out some or all of the method steps outlined in this document.

[0045] According to another aspect, a storage medium is described. This storage medium may contain software programs adapted to execute on a processor and, when implemented on a computing device, to perform some or all of the method steps outlined in this document.

[0046] As those skilled in the art will understand, the description of any of the foregoing aspects or embodiments thereof also applies to the various other aspects or embodiments thereof. For the sake of brevity, repetition of such descriptions for each aspect or embodiment has been omitted.

[0047] The methods and apparatuses, including preferred embodiments thereof as outlined in this document, may be used alone or in combination with other methods and systems disclosed herein. Furthermore, all aspects of the methods and apparatuses outlined in this document may be combined in any manner. In particular, the features of the claims may be combined with each other in any way.

[0048] Method steps and equipment features can be interchanged in many ways. In particular, as those skilled in the art will understand, the details of the disclosed methods can be implemented as part or all of the equipment suitable for performing the steps of the methods, and vice versa. Attached Figure Description

[0049] The invention is explained below by way of example with reference to the accompanying drawings, in which:

[0050] Figure 1 This is a flowchart illustrating an example of a layered coding method according to an embodiment of the present disclosure;

[0051] Figure 2 This is a block diagram schematically illustrating an example of an encoder level according to an embodiment of the present disclosure;

[0052] Figure 3 This is a flowchart illustrating an example of a method for decoding a compressed sound representation of a sound or sound field encoded as multiple hierarchical layers according to embodiments of the present disclosure;

[0053] Figure 4A and Figure 4B This is a block diagram schematically illustrating an example of a decoder level according to an embodiment of the present disclosure;

[0054] Figure 5 This is a block diagram schematically illustrating an example of a hardware implementation of an encoder according to embodiments of the present disclosure; and

[0055] Figure 6 This is a block diagram schematically illustrating an example of a hardware implementation of a decoder according to an embodiment of the present disclosure. Detailed Implementation

[0056] First, the compressed sound (or sound field) representation (hereinafter referred to as compressed sound representation for brevity) to which the method and encoder / decoder according to this disclosure can be applied will be described. Typically, a fully compressed sound (or sound field) representation (hereinafter referred to as fully compressed sound representation for brevity) may comprise (e.g., consist of) the following three components: a basic compressed sound (sound field) representation (hereinafter referred to as basic compressed sound representation for brevity), basic auxiliary information, and enhanced auxiliary information.

[0057] A basic compressed audio representation contains several components (e.g., supplementary components) (e.g., composed of them). The basic compressed audio representation may occupy a particular maximum percentage of the fully compressed audio representation. The basic compressed audio representation may consist of a sequence of coefficients representing the original HOA representation or a mono transmission signal of the primary audio signal.

[0058] Basic auxiliary information is required for decoding the basic compressed audio representation and can be assumed to be much smaller than the basic compressed audio representation. Its largest portion may consist of disjoint parts, each specifying the decompression of only one particular component of the basic compressed audio representation. Basic auxiliary information may include a first part that can be considered independent basic auxiliary information and a second part that can be considered additional basic auxiliary information.

[0059] Both the first and second parts (independent basic auxiliary information and additional basic auxiliary information) can specify the decompression of specific components of the basic compressed sound representation. The second part is optional and can be omitted. In this case, the compressed sound representation may be referred to as including the first part (e.g., basic auxiliary information).

[0060] The first part (e.g., basic auxiliary information) may contain auxiliary information describing the individual (supplementary) components of the basic compressed sound representation, independent of the other (supplementary) components. Specifically, the first part (e.g., basic auxiliary information) may individually specify the decoding of one or more of a plurality of components, independent of the other components. Therefore, the first part may be referred to as independent basic auxiliary information.

[0061] The second (optional) part may contain auxiliary information, also considered as supplementary basic auxiliary information, which may depend on other (supplementary) components describing the individual (supplementary) components of the basic compressed sound representation. This second part may also be referred to as dependent basic auxiliary information. This dependency may specifically have the following properties:

[0062] - The dependency of each volume (supplementary) component in the basic compressed sound representation is preserved to the greatest extent possible when no other specific (supplementary) components are included in the basic compressed sound representation.

[0063] - In the case of adding additional specific (supplementary) components to the basic compressed sound representation, the dependent basic auxiliary information for the individual (supplementary) components under consideration can be reduced to a subset of the original dependent basic auxiliary information, thus reducing its size.

[0064] Enhancement information is also optional. It can be used to improve or enhance (e.g., parametrically improve or enhance) the basic compressed sound representation. It can also be assumed that its size is much smaller than the size of the basic compressed sound representation.

[0065] Therefore, in embodiments, a compressed sound representation may include a basic compressed sound representation comprising multiple components, basic auxiliary information for decoding (e.g., decompressing) the basic compressed sound representation into a basic reconstructed sound representation of a sound or sound field, and enhancement auxiliary information including parameters for improving or enhancing (e.g., parametrically improving or enhancing) the basic reconstructed sound representation. The compressed sound representation may further include additional basic auxiliary information for decoding (e.g., decompressing) the basic compressed sound representation into a basic reconstructed sound representation, which may include information specifying one or more of the multiple components that depends on the decoding of the other components.

[0066] An example of this type of fully compressed sound representation is given by the compressed high-order high-fidelity stereo reproduction (HOA) sound field representation specified in Chapter 12 and Annex C.5 of the initial version of the MPEG-H 3D audio standard (Reference 1). That is, the compressed sound representation can correspond to the compressed HOA sound (or sound field) representation of a sound or sound field.

[0067] For this example, the basic compressed sound field representation (basic compressed sound representation) may contain several components (e.g., may be identified by several components). These components may be (e.g., corresponding to) a mono signal. The mono signal may be a quantized mono signal. The mono signal may represent a sequence of coefficients of the ambient HOA sound field components or the primary sound signal.

[0068] Basic auxiliary information can describe, in particular, how each of these mono signals contributes spatially to the sound field. For example, basic auxiliary information can specify the primary sound signal as a purely directional signal, meaning a general plane wave with a specific incident direction. Alternatively, basic auxiliary information can specify the mono signal as a sequence of coefficients in the original HOA representation with a specific index. As indicated above, basic auxiliary information can be further divided into a first part and a second part.

[0069] The first part is auxiliary information (e.g., independent basic auxiliary information) related to a specific individual mono signal. This independent basic auxiliary information is independent of the presence of other mono signals. For example, such auxiliary information may specify a mono signal to represent a directional signal with a specific incident direction (e.g., indicating a general plane wave). Alternatively, the mono signal may be specified as a sequence of coefficients of the original HOA representation with a specific index. The first part may be referred to as independent basic auxiliary information. Typically, the first part (e.g., basic auxiliary information) may individually specify the decoding of one or more of a plurality of mono signals, independent of other mono signals.

[0070] The second part is auxiliary information (e.g., additional basic auxiliary information) related to a specific individual mono signal. This auxiliary information depends on the presence of other mono signals. This auxiliary information can be used if the mono signal is specified as a vector-based signal (see, for example, Reference 1, Section 12.4.2.4.4). These signals are directionally distributed within the sound field, where this directional distribution can be specified by a vector. In a certain mode (see, for example, CodedVVecLength = 1), a particular component of this vector is implicitly set to zero and is not part of the compressed vector representation. These components are those with indices equal to the indices of the coefficient sequence in the original HOA representation and are part of the basic compressed sound representation. This means that if the components of the vector are encoded, their total number can depend on the basic compressed sound representation. In particular, this total number can depend on the coefficient sequence contained in the original HOA representation.

[0071] If the coefficient sequence of the original HOA representation is not included in the basic compressed sound representation, the basic auxiliary information for the dependencies of each vector-based signal consists of all vector components and has its maximum size. In the case where the coefficient sequence of the original HOA representation with certain indices is added to the basic compressed sound representation, the vector components with those indices are removed from the auxiliary information for each vector-based signal, thereby reducing the size of the basic auxiliary information for the dependencies of the vector-based signal.

[0072] Enhanced auxiliary information (e.g., enhanced auxiliary information) may include parameters related to (broadband) spatial prediction (see Reference 1, Section 12.4.2.4.3) and / or parameters related to sub-band directional signal synthesis and parameter environment replication.

[0073] Parameters related to (broadband) spatial prediction can be used to predict missing parts of the sound field (linearly) from directional signals.

[0074] Subband direction signal synthesis and parametric environment replication are compression tools recently introduced into the MPEG-H 3D audio standard through revisions [see Reference 2, Section 1]. These two tools allow for the prediction of frequency-dependent parameters of an additional mono signal to be spatially distributed, supplementing spatially incomplete or insufficient compressed HOA representations. This prediction can be based on a sequence of coefficients from the basic compressed sound representation.

[0075] Importantly, it should be noted that the supplementary contribution to the sound field mentioned above is not represented by an additional quantization signal in the compressed HOA representation, but rather by additional auxiliary information with a relatively small size. Therefore, the two tools mentioned are particularly suitable for compression of HOA representations at low data rates.

[0076] A second example of a compressed representation of one or more mono signals having the structure mentioned above may include coded spectral information for disjoint frequency bands up to a certain high frequency, which may be regarded as a basic compressed representation; basic auxiliary information specifying (e.g., by the number and width of the coded frequency bands) the coded spectral information; and enhanced auxiliary information including parameters (e.g., composed of) spectral band replication (SBR), which describes how to parametrically reconstruct spectral information for higher frequency bands not considered in the basic compressed representation from the basic compressed representation.

[0077] This disclosure proposes a layered encoding and decoding method for a fully compressed sound (or sound field) representation having the structure mentioned above.

[0078] Compression can be frame-based in the sense that it provides a compressed representation (in the form of data packets or equivalent frame payloads) for consecutive time intervals. Time intervals can have equal or different sizes. It can be assumed that these data packets contain validity flags, values ​​indicating their size, and the actual compressed representation data. In the following, not as a limitation, it will be assumed that compression is frame-based. Furthermore, not as a limitation and unless otherwise indicated, the processing of individual frames will be considered, and therefore frame indices will be omitted.

[0079] Each frame payload of the fully compressed audio (or sound field) representation under consideration is assumed to contain J data packets (or frame payloads), each data packet (or frame payload) representing one component of the basic compressed audio representation, which are represented using BSRC. j Let j = 1, ..., J be the denoting integers. Furthermore, it is assumed to contain elements using BSI. I The packet represents a component with independent basic auxiliary information (BAR), which is independent of other components specifying a particular component of the basic compressed sound representation (BSRC). j Optionally, it is also assumed to include BSI. D The package represents a packet with dependent basic auxiliary information (additional basic auxiliary information) that depends on other components specifying a particular component of the basic compressed sound representation, BSRC. j .

[0080] Two data packets BSI I and BSI D The information contained therein can optionally be grouped into a single packet (BSI) of Basic Ancillary Information. A single packet (BSI) may specifically be referred to as containing J parts, each of which specifies a particular component (BSRC) in the Basic Compressed Sound Representation. j Each of these parts can then be referred to as a part containing independent auxiliary information, and optionally a part containing dependent auxiliary information.

[0081] Finally, it may include an enhancement auxiliary information payload (enhanced auxiliary information) represented by ESI, which describes how to improve or enhance the sound (or sound field) reconstructed from a fully basic compressed representation.

[0082] The proposed layered encoding / decoding scheme solves the steps required to implement both the compression part (including packet packing for transmission) and the receiver and decompression part. Each part will be described in detail below.

[0083] First, compression and packing (e.g., for transmission) will be described. In particular, the components and elements of a fully compressed sound (or sound field) representation in the case of layered encoding and decoding will be described.

[0084] Figure 1 A flowchart illustrating an example of a compression and packing method (e.g., an encoding method, or a layered encoding method for compressed sound representation of a sound or sound field) is shown schematically. The allocation (e.g., dispatch) of individual payloads to the base layer and (M-1) enhancement layers can be accomplished by a transport layer packer. Figure 2 A block diagram illustrating an example of individual payload allocation / dispatch is shown.

[0085] As indicated above, for example, a fully compressed sound representation 2100 may relate to a compressed HOA representation that includes a basic compressed sound representation. The fully compressed sound representation 2100 may include multiple components (e.g., a mono signal) 2110-1,...2110-J, independent basic auxiliary information (basic auxiliary information) 2120, optional enhancement auxiliary information (enhanced auxiliary information) 2140, and optional dependent basic auxiliary information (additional basic auxiliary information) 2130. The basic auxiliary information 2120 may be information used to decode the basic compressed sound representation into a basic reconstructed sound representation of a sound or sound field. The basic auxiliary information 2120 may include information that independently specifies the decoding of one or more components (e.g., a mono signal) without relation to other components. The enhancement auxiliary information 2140 may include parameters for improving (e.g., enhancing) the basic reconstructed sound representation. The additional basic auxiliary information 2130 may be (additional) information used to decode the basic compressed sound representation into a basic reconstructed sound representation and may include information specifying the decoding of one or more of the multiple components dependent on the respective other components.

[0086] Figure 2The basic assumption is illustrated by the existence of multiple hierarchical layers comprising a base layer (base layer) and one or more (hierarchical) enhancement layers. For example, there can be a total of M layers, i.e., one base layer and M-1 enhancement layers. The multiple hierarchical layers have ascending layer indices. The lowest value of the layer index (e.g., layer index 1) corresponds to the base layer. It should be further understood that these layers are ordered from the base layer, through the enhancement layers, up to the overall highest enhancement layer (i.e., the overall highest layer).

[0087] The proposed method can be implemented on a frame-by-frame basis (i.e., frame-by-frame). Specifically, the compressed audio representation 2100 can be compressed over consecutive time intervals (e.g., time intervals of equal size). Each time interval can correspond to a frame. The steps described below can be implemented over each consecutive time interval (e.g., frame).

[0088] exist Figure 1 In S1010 The multiple components 2110 are subdivided into multiple component groups. Each of the multiple groups is then assigned (e.g., added or dispatched) to a corresponding layer in the multiple hierarchical layers. The number of groups corresponds to the number of layers. For example, the number of groups may be equal to the number of layers, such that each layer has one component group. As indicated above, the multiple layers may include a base layer and one or more (e.g., M-1) hierarchical enhancement layers.

[0089] In other words, the basic compressed sound representation is subdivided into parts to be assigned to each layer. Without loss of generality, the grouping can be done by M+1 J... m The description is as follows: m = 0, ..., M, where J0 = 1 and J M =J+1, such that for J m-1 ≤j <J m , BSRC j It is assigned to the m-th layer.

[0090] exist S1020 The component groups are assigned to their corresponding layers. S1030 Add (e.g., assign) basic auxiliary information 2120 to the base layer (i.e., the lowest layer among multiple hierarchical layers).

[0091] That is, due to its small size, it is proposed to include the complete basic auxiliary information (basic auxiliary information and optional additional basic auxiliary information) into the basic layer to avoid its unnecessary fragmentation.

[0092] If the compressed sound representation under consideration contains dependent basic auxiliary information (additional basic auxiliary information), the method may further include (not shown in...) Figure 1The additional basic auxiliary information is decomposed into multiple parts 2130-1, ..., 2130-M. These parts can then be added (e.g., assigned) to the base layer. In other words, parts of the additional basic auxiliary information can be included in the base layer. Each part of the additional basic auxiliary information may correspond to a corresponding layer and may include information specifying the decoding of one or more components assigned to that corresponding layer that depends on other components assigned to that corresponding layer and any layers below it.

[0093] Therefore, in the independent basic auxiliary information (BSI) I (Basic Auxiliary Information) 2120 While maintaining the allocation unchanged, the dependency basic auxiliary information must be specially processed for layered encoding and decoding to allow correct decoding at the receiver side while reducing the size of the dependency basic auxiliary information to be transmitted. It is proposed to decompose the dependency basic auxiliary information into BSI... D,m The m = 1, ..., M indicates M parts (components), where it is assumed that there is optional dependency basic auxiliary information for the compressed sound representation under consideration, and the m-th part contains the BSRCs for each component of the basic compressed sound representation assigned to the m-th layer. j J m-1 ≤j <J m The dependency of basic auxiliary information. In the case where the corresponding dependency auxiliary information is absent, for compressed sound representation, it is assumed that some BSI D,m It's empty. The various parts of the Basic Support Information System (BSI) related to dependency. D,m It can depend on all components contained in all layers up to the m-th layer (i.e., contained in all layers j = 1, ..., m) of the BSRC. j , 1≤j <J m .

[0094] If Independent Basic Auxiliary Information Package (BSI) I Its size is so small as to be negligible that it's reasonable to keep it as a whole and add (assign) it to the base layer. Alternatively, a package BSI can also be provided. I,m The independent basic auxiliary information, m = 1, ..., M, is decomposed in a similar manner to the decomposition used for dependent basic auxiliary information. The size of the basic layers can be usefully reduced by adding (assigning) portions of the independent basic auxiliary information to the corresponding components having the basic compressed sound representation.

[0095] exist S1040 Multiple portions 2140-1, ..., 2140-M of the enhanced auxiliary information can be determined. Each portion of the enhanced auxiliary information may include parameters for improving (e.g., enhancing) the reconstructed sound representation that can be obtained from data included in the corresponding layer and any layer below the corresponding layer.

[0096] The reason for implementing this step is that, in the case of layered encoding and decoding, it is important to recognize the need to additionally calculate enhancement auxiliary information for each layer, as its intention is to enhance the initial decompressed sound (or sound field), however, this depends on the layers available for decompression. Specifically, the initial decompressed sound (or sound field) for a given highest readable layer (highest usable layer) depends on the components included in the highest readable layer and any layers below it. Therefore, compression needs to provide ESI (Enhanced Sound Injection)... m The m = 1, ..., M indicates M individual enhanced auxiliary information data packets (the enhanced auxiliary information portion), where the ESI of the m-th data packet is calculated. m The enhancement auxiliary information is used to enhance the sound (or sound field) representation obtained from all data contained in the base layer and enhancement layers with indices below m (e.g., all data contained in the m-th layer and all layers below that base m).

[0097] exist S1050 The multiple parts 2140-1, ..., 2140-M of the enhanced auxiliary information are allocated (e.g., added or dispatched) to multiple layers. Each of the multiple parts of the enhanced auxiliary information is assigned to a corresponding layer among the multiple layers. For example, each of the multiple layers includes a corresponding part of the enhanced auxiliary information.

[0098] The allocation of basic and / or enhanced auxiliary information to the corresponding layers can be indicated in the configuration information generated by the encoding method. In other words, the correspondence between basic and / or enhanced auxiliary information and each layer can be indicated in the configuration information. Additionally, the configuration information can indicate for each layer the components assigned to (e.g., included in) the basic compressed sound representation of that layer. The portion of additional basic auxiliary information included in the basic layer can still correspond to a layer different from the basic layer.

[0099] In summary, at the compression level, frame packets with the following composition, as indicated by the FRAME, are provided:

[0100] FRAME = [BSRC1 ... BSRC J BSI I BSI D，1 ... BSI D，M ESI1 ... ESI M (1)

[0101] Additionally, the BSI package can be used. I and BSI D,m Where m = 1, ..., M, they are combined into a single packet BSI. In this case, the frame data packet indicated by FRAME will have the following composition:

[0102] FRAME = [BSRC1 BSRC2 ... BSRC J BSI ESI1 ESI2 ... ESI M (2)

[0103] The order of the payloads in a frame data packet is usually arbitrary.

[0104] Then, each data packet can be grouped within a payload, which is defined as a special data packet containing validity flags, values ​​indicating their size, and the actual compressed representation of the data. The use of a payload allows for simple demultiplexing at the receiver side, providing the advantage of discarding obsolete payloads without parsing them. One possible grouping is given below:

[0105] - to each BSRC j Packets, j = 1, ..., J, are assigned (e.g., dispatched) to packets marked as Each effective payload.

[0106] - ESI the m-th Enhanced Assist Information data packet m and the m-th dependency auxiliary information data packet BSI D,m Distribute (e.g., assign) to by m = 1, ..., M indicates an enhanced payload.

[0107] - Integrate Basic Support Information (BSI) I Package assigned by The indicated separate auxiliary information payload.

[0108] Optionally, if the size of the independent basic auxiliary information is large, its components (BSI) can be used. I,m Each m-th component in m = 1, ..., M is assigned (e.g., distributed) to the enhanced payload. In this scenario, the auxiliary information payload It is empty and can be ignored.

[0109] Another option is to include all dependency basic auxiliary information data packets (BSI). D,m Assigned to auxiliary information payload This is reasonable when the size of the dependent basic auxiliary information is small.

[0110] Finally, a frame data packet marked with FRAME can be provided, consisting of the following components:

[0111]

[0112] The order of the payloads in a frame data packet is usually arbitrary.

[0113] The method may further include (not shown in) Figure 1 For each of the multiple layers, a transport layer packet is generated (e.g., a basic layer packet 2200 and M-1 enhancement layer packets 2300-1,...,2300-(M-1)), which includes data of the corresponding layer (e.g., components, basic auxiliary information and enhancement auxiliary information for the basic layer, or components and enhancement auxiliary information for the one or more enhancement layers).

[0114] Transport layer packets used for different layers can have different transmission priorities. Therefore, the method may further include (not shown in...) Figure 1 (In the middle) a transport stream is generated for the transmission of data across multiple layers, where the base layer has the highest transport priority and the hierarchical enhancement layers have decreasing transport priorities. A higher transport priority corresponds to a greater degree of error protection, and vice versa.

[0115] Unless a step requires certain other steps as prerequisites, the aforementioned steps can be performed in any order, and Figure 1 The exemplary order shown is to be understood as non-limiting.

[0116] Figure 3 A decoding method for decoding or decompressing (unpacking) a compressed sound representation (or sound field) is illustrated. Examples of corresponding receivers and decompression stages are schematically depicted. Figure 4A and Figure 4B In the block diagram.

[0117] Following the above, a compressed audio representation can be encoded across multiple hierarchical layers. Multiple layers may be assigned (e.g., may include) components of a basic compressed audio representation, which are distributed among the layers in respective component groups. The basic layer may include basic auxiliary information for decoding the basic compressed audio representation. Each layer may include one of the aforementioned portions of its enhanced auxiliary information, comprising parameters for improving the basic reconstructed audio representation, which may be derived from data included in that respective layer and any layers below it.

[0118] The proposed method can be implemented on a frame-by-frame basis (i.e., frame-by-frame). In particular, the restored representation of sound or sound field can be generated for consecutive time intervals (e.g., time intervals of equal size). For example, a time interval can be a frame. The steps described below can be implemented for each consecutive time interval (e.g., frame).

[0119] exist S3010The system receives data payloads (e.g., transmission layer packets) corresponding to multiple layers. The data payloads may be received as part of a bitstream containing a compressed HOA representation of sound or a sound field, corresponding to multiple hierarchical layers. The hierarchical layers include a base layer and one or more hierarchical enhancement layers. Multiple layers are assigned components of the base compressed sound representation of the sound or sound field. The components are assigned to each layer within a component group.

[0120] Each layer of packets can be multiplexed to provide a frame packet representing the fully compressed audio received. The received frame packet can be indicated by the following formula:

[0121]

[0122] In the package BSI I and BSI D,m In the case where m = 1, ..., M is an alternative to a single-packet BSI, the packets at each layer can be multiplexed to provide a frame packet representing the fully compressed audio received, as indicated by the following formula:

[0123]

[0124] Regarding the payload, the received frame packets can be given by the following formula:

[0125]

[0126] The received frame packets can then be transmitted to the decompressor or decoder 4100. If the transmission of a single layer is error-free, then at least the included enhanced auxiliary information payload... The validity flag of a portion (e.g., corresponding to a part of the enhanced auxiliary information) is set to "true". In the event of an error due to the transmission of a single layer, the validity flag within the enhanced auxiliary information payload of at least that layer is set to "false". Therefore, the validity of a layer packet can be determined from the validity of the included enhanced auxiliary information payload (e.g., from its validity flag).

[0127] In the decompressor 4100, the received frame packets can be demultiplexed. For this purpose, information related to the size of each payload can be used to avoid unnecessarily parsing the data of each payload.

[0128] exist S3020 The first layer index, which indicates the highest layer (e.g., the highest usable layer or the highest decodeable layer), is determined among multiple layers for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field.

[0129] In addition, S3020 The value (e.g., layer index) N of the highest layer (the highest usable layer) used for decompression in the basic sound representation can be selected.B The highest enhancement layer for decompression, which is actually used for basic sound representation, consists of N. B -1 is given. Because each layer contains exactly one augmentation auxiliary information payload (part of the augmentation auxiliary information), the validity of the containing layer (e.g., whether it is effectively received) can be determined based on the augmentation auxiliary information payload. Therefore, this selection can use all augmentation auxiliary information payloads (ESI). m m = 1, ..., M (or correspondingly, This can be accomplished by (m = 1, ..., M).

[0130] exist S3030 This yields a basic reconstructed sound representation. Using basic auxiliary information (or in general, using basic auxiliary information), the basic reconstructed sound representation can be obtained from the components assigned to the highest usable layer indicated by the first-level index and any layers below this highest usable layer.

[0131] The basic compressed sound representation components are BSRC1, ..., BSRC1. J The payload may include (all) basic auxiliary information payloads (e.g., BSI or BSI). I and BSI D,m (m = 1, ..., M) and the value N B Together, they are provided to the basic representation decompression processing unit 4200. The basic representation decompression processing unit 4200 (described in...) Figure 4A and 4B (in the middle) only use the N contained in the lowest N B Each layer (i.e., the basic layer and N) B The basic sound (or sound field) representation is reconstructed from the basic compressed sound representation components within the -1 enhancement layers (i.e., the layers up to the layer indicated by the first layer index). Alternatively, only those components contained in the lowest N can be used to reconstruct the basic sound (or sound field) representation. B The payloads of the basic compressed sound representation components in each layer, together with the corresponding basic auxiliary information payloads, are provided to the basic representation decompression processing unit 4200.

[0132] The necessary information regarding which components of the basic compressed sound (or sound field) representation are included in each layer is assumed to be known by the decompressor 4100 from a data packet containing configuration information, assuming that the data packet is sent and received before the frame data packet.

[0133] In order to provide dependency auxiliary information data packets BSI D,m m = 1, ..., N B and enhanced auxiliary information data packets All enhanced payloads can be combined with the value N E and value N B Together they are input to the partial parser 4400 of the decompressor 4100 (see Figure 4B The parser can discard all payloads and data packets that will not be used for actual decompression. If N E If the value is zero, then it is assumed that all enhanced auxiliary information packets are empty.

[0134] If the base layer includes at least one dependent basic auxiliary information payload (the portion with additional basic auxiliary information) corresponding to the corresponding layer, each dependent basic auxiliary information payload (e.g., BSI) D,m m = 1, ..., N B Decoding the portion of the additional basic auxiliary information may include (i) decoding the portion of the additional basic auxiliary information by referring to components assigned to its corresponding layer and any layer below that corresponding layer (preliminary decoding), and (ii) correcting the portion of the additional basic auxiliary information by referring to components assigned to the highest usable layer and any layer between the highest usable layer and the corresponding layer (correction). The additional basic auxiliary information corresponding to the corresponding layer includes information specifying the decoding of one or more components assigned to the corresponding layer that depends on the decoding of other components assigned to the corresponding layer and any layer below that corresponding layer.

[0135] Then, using the basic auxiliary information and the correction portion of the additional basic auxiliary information obtained from the portion corresponding to the layer up to the highest usable layer, a basic reconstructed sound representation can be obtained (e.g., generated) from the components assigned to the highest usable layer and any layer below the highest usable layer.

[0136] Specifically, each payload BSI D,m m = 1, ..., N B The initial decoding may involve utilizing its properties for the first J contained in the first m layers. m -1 basic compressed sound representation components BSRC1, ..., BSRC (Jm)-1 The dependency is assumed at the coding level.

[0137] Each payload BSI D,m m = 1, ... N B The sequential correction may involve considering the fundamental sound components ultimately being extracted from the first N... B >The first of m layers The basic compressed sound representation components are BSRC1, ... (It has more components than assumed for initial decoding) Reconstruction. Therefore, correction can be accomplished by discarding outdated information, which is made possible by the initial assumption of the nature of the dependent basic auxiliary information, which is that if some supplementary components are added to the basic compressed sound representation, the dependent basic auxiliary information for each volume (supplementary) component becomes a subset of the original information.

[0138] exist S3040 The second-level index can be determined. The second-level index can indicate one or more portions of the enhancement auxiliary information that should be used to improve (e.g., enhance) the basic reconstructed sound representation.

[0139] In addition to the first-level index, the index (second-level index) N of the enhanced auxiliary information payload (the second enhanced information portion) to be used for decompression can also be determined. E Second-level index N E But it always equals the first-level index N. B It may be equal to zero. Enhancement may always be done based on the basic sound representation obtained from the highest usable layer, or it may not be done at all.

[0140] exist S3050 The reconstructed sound representation of a sound or sound field is obtained (e.g., generated) from the basic reconstructed sound representation by referring to the second-level index.

[0141] That is, the reconstructed sound representation is obtained by (parametrically) improving or enhancing the basic reconstructed sound representation (e.g., by using enhanced auxiliary information (part of the enhanced auxiliary information) indicated by the second-level index). As further indicated below, the second-level index may indicate that no enhanced auxiliary information is used at this stage. The reconstructed sound representation then corresponds to the basic reconstructed sound representation.

[0142] To this end, the basic sound representation is reconstructed along with all the enhanced auxiliary information payloads ESI1, ..., ESI. M Basic auxiliary information payload (e.g., BSI or BSI) I and BSI D,m (m = 1, ..., M), and the value N E Together they are provided to the enhanced representation decompression processing unit 4300 (described in Figure 4A and 4B (in the middle), it only uses the enhanced auxiliary information payload. To calculate the final enhanced sound (or sound field) representation 2100', all other enhancement-aid information payloads are discarded. Alternatively, instead of all enhancement-aid information payloads, only the enhancement-aid information payloads are included. Provided to the enhanced representation decompression processing unit 4300. If N E If the value is zero, all augmentation auxiliary information payloads are discarded (or alternatively, no augmentation auxiliary information payload is provided), and the reconstructed final augmented voice representation 2100' is equal to the reconstructed basic voice representation. Augmentation auxiliary information payload This can be obtained from part of the parser 4400.

[0143] Figure 3 It also generally demonstrates decoding of compressed HOA representations based on basic auxiliary information associated with the base layer and enhanced auxiliary information associated with one or more hierarchical enhancement layers.

[0144] Unless a step requires certain other steps as prerequisites, the aforementioned steps can be performed in any order, and Figure 3 The exemplary order shown is to be understood as non-limiting.

[0145] Next, details of the layer selection (selection of the first and second layer indices) for decompression in steps S3020 and S3040 will be described.

[0146] Determining the first-level index may involve determining whether the corresponding layer has been effectively received for each layer. Determining the first-level index may further involve defining the first-level index as the layer index immediately following the lowest layer that has not been effectively received. Whether a layer has been effectively received can be determined by evaluating whether the enhanced auxiliary information payload of that layer has been effectively received. This can then be accomplished by evaluating the validity flags within the enhanced auxiliary information payload.

[0147] Determining the second-level index may typically involve determining that the second-level index is equal to the first-level index, or determining the index value that indicates no augmenting auxiliary information is used when obtaining the reconstructed sound representation as the second-level index (e.g., index value 0).

[0148] In cases where all frame data packets can be decompressed independently of each other, the highest layer (highest usable layer) that can be actually used for basic sound representation can be identified by the number N. B and the index N of the enhanced auxiliary information payload to be used for decompression. E The maximum number L of effective augmentation and assistance information payloads is set, which can be determined by evaluating the effectiveness flags within the augmentation and assistance information payloads. By utilizing knowledge of the dimensions of each augmentation and assistance information payload, complex analysis of the actual data pairs of the payloads can be avoided in order to determine the effectiveness of the payloads.

[0149] That is, if the compressed audio representation used for continuous time intervals can be decoded independently, then the second-level index can be determined to be equal to the first-level index. In this case, the reconstructed basic audio representation can be enhanced based on the enhanced auxiliary information payload of the highest usable layer.

[0150] In cases using differential decompression with inter-frame correlation, decisions from previous frames must be considered separately. It should be noted that, with differential decompression, independent frame packets are typically transmitted at regular time intervals to allow decompression to begin from these points in time, where the value N... B and N EThe determination becomes frame-independent and is performed as described above.

[0151] To explain the proposed frame-dependent decision in detail, the highest number of effective augmentation auxiliary information payloads (e.g., layer index) in the k-th frame is denoted as L(k), and the highest layer number (e.g., layer index) to be selected for decompression of the basic sound representation is denoted as N. B (k), and label the number of enhanced auxiliary information payloads (e.g., layer indexes) to be used for decompression as N. E (k).

[0152] Therefore, from N B (k) The highest layer number of the decompression used for basic sound representation, marked with a symbol, can be calculated according to the following formula.

[0153] N B (k)=min(N B (k-1), L(k)). (7)

[0154] By selecting N B (k) is not greater than N B (k-1) and L(k) ensure that all the information required for differential decompression of the basic sound representation is available.

[0155] That is, if the compressed audio representations of consecutive time intervals (e.g., frames) cannot be decoded independently of each other, determining the first layer index may include determining whether the corresponding layer has been effectively received for each layer, and determining the first layer index for a given time interval as the smaller of the first layer index of the time interval preceding that given time interval and the layer index of the layer immediately below the lowest layer that has not been effectively received.

[0156] The number N of enhanced auxiliary information payloads to be used for decompression E (k) can be determined according to the following formula:

[0157]

[0158] Among them, N E (k) Selecting 0 indicates that the reconstructed basic sound will not be improved or enhanced using augmented auxiliary information.

[0159] This specifically means that only the highest layer number N used for decompression of basic sound representation is required. B (k) If it remains unchanged, the same corresponding enhancement layer number is selected. However, in N B In the case where (k) changes, by changing N E (k) is set to zero to disable enhancement. Due to the differential compression of the assumed enhancement auxiliary information, its value is determined by N. BThe change of (k) is impossible because it would require decompression of the corresponding enhanced auxiliary information layer in the previous frame, which is assumed not to have been implemented.

[0160] That is, if the compressed audio representations used for consecutive time intervals (e.g., frames) cannot be decoded independently, determining the second-level index may involve determining whether the first-level index of a given time interval is equal to the first-level index of a prior time interval. If the first-level index of a given time interval is equal to the first-level index of a prior time interval, the second-level index of the given time interval can be determined (e.g., selected) to be equal to the first-level index of the given time interval. On the other hand, if the first-level index of a given time interval is not equal to the first-level index of a prior time interval, the index value indicating that no enhancement auxiliary information is used when obtaining the reconstructed audio representation can be determined (e.g., selected) as the second-level index.

[0161] Alternatively, if decompression will have up to N... E If all the enhanced auxiliary information payloads of number (k) are decompressed in parallel, then the selection rule in equation (4) can be replaced by the following:

[0162] N E (k)= N B (k) (9)

[0163] Finally, it was pointed out that for differential compression, the highest layer number N is used. B It can increase only in individual frame packets, but may decrease in each frame.

[0164] It should be understood that the proposed layered coding method for compressed sound representation can be implemented by an encoder for layered coding of compressed sound representation. Such an encoder may include units suitable for performing the above steps. An example of such an encoder 5000 is schematically depicted in... Figure 5 For example, such an encoder 5000 may include a component subdivision unit 5010 suitable for implementing S1010 mentioned above, a component allocation unit 5020 suitable for implementing S1020 mentioned above, a basic auxiliary information allocation unit 5030 suitable for implementing S1030 mentioned above, an enhanced auxiliary information partitioning unit 5040 suitable for implementing S1040 mentioned above, and an enhanced auxiliary information allocation unit 5050 suitable for implementing S1050 mentioned above. It should also be understood that each unit of such an encoder may be embodied by a processor 5100 of a computing device, which is suitable for implementing the processing performed by each of the units, i.e., suitable for implementing some or all of the steps mentioned above, and any further steps of the proposed encoding method. The encoder or computing device may further include a memory 5200 accessible by the processor 5100.

[0165] It should be understood that the proposed method for decoding compressed audio representations encoded in multiple hierarchical layers can be implemented by a decoder for decoding compressed audio representations encoded in multiple hierarchical layers. Such a decoder may include units suitable for performing the steps described above. An example of such a decoder 6000 is schematically depicted in... Figure 6 For example, such a decoder 6000 may include a receiving unit 6010 suitable for implementing S3010 mentioned above, a first-layer index determination unit 6020 suitable for implementing S3020 mentioned above, a basic reconstruction unit 6030 suitable for implementing S3030 mentioned above, a second-layer index determination unit 6040 suitable for implementing S3040 mentioned above, and an enhanced reconstruction unit 6050 suitable for implementing S3050 mentioned above. It should also be understood that each unit of such a decoder may be embodied by a processor 6100 of a computing device, which is suitable for implementing the processing performed by each of the units, i.e., suitable for implementing some or all of the steps mentioned above, and any further steps of the proposed decoding method. The decoder or computing device may further include a memory 6200 accessible by the processor 6100.

[0166] It should be noted that the specification and accompanying drawings only describe the principles of the proposed methods and apparatus. Therefore, it should be understood that those skilled in the art will be able to conceive of various configurations embodying the principles of the invention and encompassing its spirit and scope, even if they are not explicitly described or shown herein. Furthermore, all examples set forth herein are, in principle, explicitly intended for illustrative purposes only, to assist the reader in understanding the principles and concepts of the proposed methods and apparatus provided by the inventors for further development of the technology, and should be interpreted as not being limited to the examples and conditions specifically stated herein. Moreover, the descriptions of the principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to cover their equivalents.

[0167] The methods and apparatus described in this document can be implemented as software, firmware, and / or hardware. Some components may be implemented, for example, as software operating on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or application-specific integrated circuits. Signals encountered in the described methods and apparatus may be stored in a medium, such as random access memory or optical storage media. They may be transmitted via a network, such as a radio network, satellite network, wireless network, or wired network, such as the Internet.

[0168] Reference document 1: ISO / IEC JTC1 / SC29 / WG11 23008-3:2015(E). Information technology - High efficiency coding and media delivery in heterogeneous environments – Part 3: 3D audio, February 2015.

[0169] Reference document 2: ISO / IEC JTC1 / SC29 / WG11 23008-3:2015 / PDAM3. Information technology - High efficiency coding and media delivery in heterogeneous environments – Part 3: 3D audio, AMENDMENT 3: MPEG-H 3D Audio Phase 2, July 2015.

Claims

1. A method for decoding a compressed high-order high-fidelity stereo reproduction (HOA) sound representation of a sound or sound field encoded into multiple hierarchical layers using layered coding, the method comprising: Receive a bitstream containing the compressed HOA representation, the compressed HOA representation corresponding to a plurality of hierarchical layers including a base layer and at least one enhancement layer, wherein at least one of the plurality of hierarchical layers includes a component of the basic compressed sound representation of the sound or sound field, the component corresponding to a plurality of mono signals. It is determined that the parameter CodedVVecLength is not equal to 1, and based on this, it is determined that all components of the vector corresponding to the compressed HOA representation are provided; and The compressed HOA representation is decoded based on basic auxiliary information associated with the base layer and enhanced auxiliary information associated with the enhancement layer, wherein the basic auxiliary information indicates that at least one individual mono signal represents a directional signal with an incident direction, and wherein the enhanced auxiliary information includes information that allows prediction of missing portions of the sound or sound field.

2. The method of claim 1, wherein the enhanced auxiliary information includes parameters relating to at least one of: spatial prediction, sub-band direction signal synthesis, and parameter environment replication.

3. A non-transitory computer-readable storage medium, comprising instructions that, when executed by a processor, cause the method according to claim 1 or 2 to be performed.

4. An apparatus for decoding a compressed high-fidelity stereo reproduction (HOA) sound representation of a sound or sound field encoded into multiple hierarchical layers using layered coding, the apparatus comprising: A receiver is configured to receive a bitstream containing the compressed HOA representation, the compressed HOA representation corresponding to a plurality of hierarchical layers including a base layer and at least one enhancement layer, wherein the plurality of hierarchical layers include components of the basic compressed sound representation of the sound or sound field, the components corresponding to a plurality of mono signals. The processor determines that the parameter CodedVVecLength is not equal to 1, and based on this, determines that all components of the vector corresponding to the compressed HOA representation are provided, and A decoder is used to decode the compressed HOA representation based on basic auxiliary information associated with the base layer and enhanced auxiliary information associated with the enhancement layer, wherein the basic auxiliary information indicates that at least one individual mono signal represents a directional signal with an incident direction, and wherein the enhanced auxiliary information includes information that allows prediction of missing portions of the sound or sound field.

5. The device of claim 4, wherein the enhanced auxiliary information includes parameters relating to at least one of the following: spatial prediction, sub-band direction signal synthesis, and parameter environment replication.

6. An apparatus for decoding a compressed high-fidelity stereo reproduction (HOA) sound representation of a sound or sound field encoded into multiple hierarchical layers using layered coding, comprising: processor, and A non-transitory computer-readable storage medium, comprising instructions that, when executed by the processor, cause the method according to any one of claims 1-2 to be performed.

7. A computer program product having instructions that, when executed by a processor, cause the method according to any one of claims 1-2 to be performed.

Images