Method and apparatus for improving coding of side information required for coding higher order ambisonics representation of sound field

Abstract

To improve coding of side information required for coding a higher order ambisonics representation of a sound field.SOLUTION: Higher order ambisonics represents three-dimensional sound independent of a specific loudspeaker set-up. However, transmission of a HOA representation results in a very high bit rate. Therefore, compression with a fixed number of channels is used, in which directional and ambient signal components are processed differently. For coding, various portions of an original HOA representation are predicted from the directional signal components. This prediction provides side information which is required for corresponding decoding. By using some additional specific purpose bits, known side information coding processing is improved in that the required number of bits for coding that side information is reduced on average.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for improving the coding of side information required for coding a higher order Ambisonics representation of a sound field. [Background technology]

[0002] Higher Order Ambisonics (HOA) offers one possibility for representing three-dimensional sound, among other techniques such as wave field synthesis (WFS) or channel-based approaches such as 2.2 multichannel audio formats. In contrast to channel-based methods, the HOA representation offers the advantage of being independent of a specific loudspeaker setup. However, this flexibility comes at the cost of a decoding process required for the reproduction of the HOA representation on a specific loudspeaker setup. Compared to the WFS approach, where the number of speakers required is usually much higher, the HOA signal may be rendered to setups with only a few speakers. A further advantage of HOA is that the same representation can also be used for binaural rendering to headphones without modification.

[0003] HOA is based on the representation of the spatial density of complex harmonic plane wave amplitudes by a truncated spherical harmonic (SH) expansion. Each expansion coefficient is a function of angular frequency, which can be equivalently represented by a time-domain function. Thus, without loss of generality, the complete HOA sound field representation can be assumed to actually consist of O time-domain functions, where O denotes the number of expansion coefficients. These time-domain functions are referred to in the following as HOA coefficient sequences or HOA channels, although they are equivalent.

[0004] The spatial resolution of the HOA representation improves with increasing maximum order of the expansion, N. Unfortunately, the number of expansion coefficients, O, is quadratic with the order, N, specifically, O=(N+1). 2For example, a typical HOA representation using order N=4 requires O=25 HOA (expansion) coefficients. According to the considerations made earlier, the total bit rate for the transmission of the HOA representation grows in the form of S and the number of bits per sample, N b Given that, O·f S N b As a result, we define the HOA expression of order N=4 as f S = N samples per sample at a sampling rate of 48 kHz b = 16 bits leads to a bit rate of 19.2 MBits / s, which is too high for many practical applications, e.g. streaming. Thus, compression of the HOA representation is highly desirable.

[0005] Compression of HOA sound field representations has been proposed in WO2013 / 171083A1, EP13305558.2 and PCT / EP2013 / 075559. These processes have in common that they perform a sound field analysis and decompose a given HOA representation into a directional component and a residual ambient component. On the one hand, it is assumed that the final compressed representation consists of several quantized signals resulting from a perceptual coding of the directional signal and the associated coefficient sequence of the ambient HOA component. On the other hand, it is assumed that the final compressed representation contains additional side information related to the quantized signals. This side information is necessary for the reconstruction of the HOA representation from its compressed version.

[0006] An important part of the side information is the description of the prediction of parts of the original HOA representation from the directional signal, which is referred to below as the spatial prediction, since for this prediction it is assumed that the original HOA representation is equivalently represented by several spatially dispersed general plane waves incident from spatially uniformly distributed directions.

[0007] Coding of such side information related to spatial prediction is described in [1]. However, this state-of-the-art coding of side information is rather inefficient. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] ISO / IEC JTC1 / SC29 / WG11, N14061, "Working Draft Text of MPEG-H 3D Audio HOA RM0", November 2013, Geneva, Switzerland Summary of the Invention [Problem to be solved by the invention]

[0009] The problem to be solved by the present invention is to provide a more efficient way of coding such spatial prediction related side information. [Means for solving the problem]

[0010] This problem is solved by the methods disclosed in claims 1 and 6. Apparatuses utilizing these methods are disclosed in claims 2 and 7.

[0011] Encoded side information representation data ζ COD A bit is added before ζ. This bit tells whether any prediction should be performed or not. This feature is CODThis reduces the average bit rate for the transmission of data over time. Furthermore, in individual situations, it is more efficient to transmit or transfer the number of active predictions and their respective indexes, instead of using a bit array indicating for each direction whether prediction is performed or not. A single bit can be used to indicate how the index of the direction in which prediction should be performed is coded. On average, this operation reduces the ζ COD The bit rate for transmission of data is further reduced.

[0012] In principle, the method of the invention is suitable for improving the coding of side information required for coding a Higher Order Ambisonics representation (HOA) of a sound field having an input time frame of a sequence of HOA coefficients, where a dominant directional signal and a residual ambient HOA component are determined and a prediction is used for said dominant directional signal, thereby providing, for a coded frame of HOA coefficients, side information data describing said prediction, said side information data comprising: A bit array indicating whether prediction is performed for a direction; A bit array where each bit indicates a type of prediction for the direction in which the prediction should be performed; A data array with elements representing the indices of the directional signals to be used for the forecast to be performed; a data array having elements representing quantized scaling factors; The method involves: providing a bit value indicating whether the prediction should be performed; omitting the bit array and the data array in the side information data if no prediction is to be performed; If the prediction should be performed, providing a bit value indicating whether or not a data array containing the number of active predictions and an index of the direction in which the prediction should be performed is included in the side information data instead of the bit array indicating whether or not a prediction is performed for a direction.

[0013] In principle, the inventive device is suitable for improving the coding of side information required for coding a Higher Order Ambisonics representation (HOA) of a sound field having an input time frame of a sequence of HOA coefficients, where a dominant directional signal and a residual ambient HOA component are determined and a prediction is used for said dominant directional signal, thereby providing, for a coded frame of HOA coefficients, side information data describing said prediction, said side information data comprising: A bit array indicating whether prediction is performed for a direction; A bit array where each bit indicates a type of prediction for the direction in which the prediction should be performed; A data array with elements representing the indices of the directional signals to be used for the forecast to be performed; a data array having elements representing quantized scaling factors; The device: providing a bit value indicating whether the prediction should be performed; omitting the bit array and the data array in the side information data if no prediction is to be performed; means for providing a bit value indicating whether, if the prediction should be performed, a data array including the number of active predictions and an index of the direction in which the prediction should be performed is included in the side information data instead of providing the bit array indicating whether a prediction is performed for a direction or not.

[0014] Advantageous further embodiments of the invention are disclosed in the respective dependent claims. [Brief description of the drawings]

[0015] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings. [Figure 1] FIG. 1 illustrates an exemplary encoding of side information related to spatial prediction in the HOA compression process described in EP13305558.2. [Diagram 2]FIG. 1 illustrates an exemplary decoding of side information related to spatial prediction in the HOA decompression process described in patent application EP13305558.2. [Diagram 3] FIG. 1 illustrates the HOA decomposition described in patent application PCT / EP2013 / 075559. [Figure 4] FIG. 1 shows the directions of general plane waves (drawn as crosses) representing the residual signal and the directions of dominant sound sources (drawn as circles). The directions are presented in a three-dimensional coordinate system as sampling positions on a unit sphere. [Diagram 5] FIG. 1 illustrates state-of-the-art coding of spatial prediction side information. [Figure 6] FIG. 1 illustrates the inventive coding of side information for spatial prediction. [Figure 7] FIG. 2 illustrates the inventive decoding of encoded spatial prediction. [Figure 8] Continued from Figure 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In the following, the HOA compression and decompression process described in patent application EP13305558.2 is summarized in order to provide a context in which the inventive coding of side information related to spatial prediction is used.

[0017] <HOA Compression> In Fig. 1 it is shown how the coding of side information related to spatial prediction can be embedded in the HOA compression process described in patent application EP13305558.2. For the HOA representation compression a frame-by-frame process is assumed with non-overlapping input frames C(k) of the HOA coefficient sequence of length L, where k stands for the frame index. The first stage or step 11 / 12 in Fig. 1 is optional and involves the kth and (k-1)th non-overlapping frames of the HOA coefficient sequence C(k) being compressed into long frames L.

number

[0018] The long frames C(k) are successively used in stage or step 13 for the estimation of the dominant sound source direction as described in EP13305558.2. This estimation is based on a data set of indices of the detected relevant directional signals.

number

number

[0019] In step or stage 14, the current (long) frame of the HOA coefficient sequence (C(k)) is aggregated (as proposed in EP13305156.5).

number

number

[0020] In step or stage 15, the ambient HOA component C AMB The number of coefficients in (k-2) is only O RED +DN DIR,ACT The sequence is reduced to include (k-2) non-zero HOA coefficients, where

number

number

[0021] Reduced number O RED +N DIR,ACTThe final ambient HOA representation with (k-2) non-zero coefficient sequences is C AMB,RED The index of the selected surrounding HOA coefficient sequence is denoted by (k-2).

number

number

[0022] According to the invention, after the decomposition of the original HOA representation in step / stage 14, the spatial prediction parameters or side information data ζ(k-2) resulting from the decomposition of the HOA representation are converted in step or stage 19 into the coded data representation ζ COD To provide the (k-2) index set

number

[0023] <HOA decompression> In FIG. 2, the received encoded side information data ζ related to spatial prediction COD It is exemplarily shown how the decoding of (k-2) is embedded in the HOA decompression process described in FIG. 3 of patent application EP13305558.2 in step or stage 25. CODThe decoding of (k-2) is performed by inputting the decoded version ζ(k-2) into the synthesis of the HOA representation in stage or step 23 using the received index set

number

[0024] In step or stage 21,

number

number

[0025] In the signal redistribution stage or stage 22

number

number

number

number

[0026] In the synthesis stage or stage 23, the current frame of the desired full HOA representation is

number

number

number

number

number

[0027] Number 22 is the component in PCT / EP2013 / 075559

number

number

number

number

number

number

number

[0028] <HOA decomposition> In relation to Fig. 3, the HOA decomposition process is detailed in order to explain the meaning of spatial prediction therein. The process is derived from the process described in relation to Fig. 3 of patent application PCT / EP2013 / 075559.

[0029] First, the smoothed directional signal X DIR (k-1) and its HOA representation C DIR (k-1) is the long frame of the input HOA representation in step or stage 31.

number

number

number

number

[0030] In step 33, the original HOA representation [C(k-1) with a tilde] and the HOA representation C of the dominant directional signal are DIR The residual between (k-1) and O directional signals

number

[0031] In stage or step 34, these directional signals are combined to produce a dominant directional signal X DIR Predicted from (k-1). Predicted signal

number

number

[0032] In step 35, the predicted directional signal

number

number

[0033] In step or stage 37, the original HOA representation (C(k-2)) and the HOA representation C(k-2) of the dominant directional signal are compared. DIR HOA representation of predicted directional signals from (k-2) uniformly distributed directions

number

[0034] The required signal delays in the processing of FIG.

[0035] Spatial Prediction The goal of spatial prediction is to obtain O residual signals

number

number

[0036] Each residual signal

number

[0037] Each directional signal

number

[0038] To illustrate the meaning of spatial prediction by an example, consider the decomposition of an HOA representation of order N=3. Here, the maximum number of directions to be extracted is equal to D=4. For simplicity, we further assume that only directional signals with indexes 1 and 4 are active, while directional signals with indexes 2 and 3 are inactive. Furthermore, for simplicity, we assume that the dominant sound source directions are constant for the frames considered, i.e., for d=1,4, Ω ACT,d (k-3)=Ω ACT,d (k-2)=Ω ACT,d (k-1)=Ω ACT,d (k)=Ω ACT,d (5) It is assumed that the order is N=3, resulting in a spatially dispersed general plane wave.

number

[0039] State-of-the-art parameters for describing spatial prediction One way of describing spatial prediction is presented in the above-mentioned ISO / IEC non-patent document 1. In that document, the signal

number

[0040] ·Element p TYPE,qA vector p consisting of (k-1), q=1,…,O TYPE (k-1) is the qth direction Ω q Indicates whether a prediction is performed for a given item, and if so, what type of prediction. The meaning of the above elements is as follows: p TYPE,q (k-1)=0 direction Ω q If there is no prediction =1 directionΩ q For full-bandwidth prediction (6) =2 directionΩ q For low-frequency prediction.

[0041] ·Element p IND,d,q (k-1), d=1,…,D PRED , q=1,…,O IND (k-1) is the direction Ω from the corresponding directional signal q represents the index at which a prediction for q If there is no prediction to be performed on the matrix P IND The corresponding column of (k-1) consists of 0s. Furthermore, q The directional signal used for prediction of PRED If it is less than 1, P IND The unnecessary elements in the q-th column of (k-1) are also zero.

[0042] The corresponding quantized predictor p Q,F,d,q (k-1), d=1,…,D PRED , matrix P with q=1,…,O Q,F (k-1).

[0043] The following two parameters must be known at the decoder to allow proper interpretation of these parameters: ·General plane wave signal

number

[0044] These two parameters need to be set to fixed values ​​known to the encoder and decoder, or they need to be transmitted additionally, but significantly less frequently than the frame rate. The latter option may be used to adapt the two parameters to the HOA representation to be compressed. An example for a parameter set is O=16, D PRED =2, B SC =8, it could look something like this:

[0045]

number

number

number

number

number

[0046] Given this side information, the prediction is assumed to be performed as follows.

[0047] First, the quantized predictor p Q,F,d,q (k-1), d=1,…,D PRED , q=1,…,O are dequantized to give the actual predictors.

[0048]

number

[0049] In the above example, B SC = 8, the resulting dequantized predictor vector is:

[0050]

number

[0051] Signal predicted as signal

number

number

number

number

[0052] As already stated, and now seen from equation (17), the signal

number

[0053] State-of-the-art coding of side information related to spatial prediction The coding of side information for spatial prediction is addressed in the above mentioned ISO / IEC non-patent document 1. It is summarized in Algorithm 1, depicted in Fig. 5 and explained below. For a clearer presentation, the frame index k-1 is ignored in all equations.

[0054] First, a bit array ActivePred is generated that consists of O bits, where the bits ActivePred[q] are in the direction Ω q Indicates whether prediction is performed for. The number of "1"s in this array is represented by NumActivePred.

[0055] Then, a bit array PredType of length NumActivePred is generated, where each bit indicates the type of prediction, i.e. full-band or low-pass, for the direction in which the prediction should be performed. At the same time, a bit array PredType of length NumActivePred D PRED An unsigned integer array PredDirSigIds is generated whose elements indicate, for each active prediction, the D PRED D REPDIf fewer directional signals are used for prediction, the index is assumed to be set to 0. Each element of the array PredDirSigIds is

number

[0056] Finally, an integer array QuantPredGains of length NumNonZeroIds is created whose elements represent the quantized scaling factors p Q,F,d,q (k-1). The corresponding dequantized scaling factor p F,d,q The dequantization to obtain (k-1) is given in equation (10). Each element of the array QuantPredGains is SC It is assumed to be represented by bits.

[0057] Finally, the encoded representation of the side information, ζ COD teeth, ζ COD =[ActivePred PredType PredDirSigIds QuantPredGains] (19) It consists of the above four sequences according to the above.

[0058] To illustrate this encoding by example, the encoded representation of equations (7) to (9) is used:

number

[0059] Coding Side Information Related to Spatial Prediction According to the Invention To increase the efficiency of coding side information related to spatial prediction, state-of-the-art processes are advantageously modified.

[0060] A) When encoding a HOA representation of a typical sound scene, we observe that there are often frames for which the HOA compression process decides not to perform any spatial prediction at all. However, in such frames, the bit array ActivePred consists only of 0s, and the number of 0s is equal to O. Since such frame content occurs quite frequently, our process optimizes the encoded representation ζ COD A single bit PSPredictionActive is added before the bit PSPredictionActive, which indicates whether any prediction should be performed or not. If the bit PSPredictionActive has the value 0 (or in the alternative "1"), the array ActivePred and further data related to the prediction are stored in the encoded side information ζ COD In practice, this process is COD , decreasing the average bit rate for transmission over time.

[0061] B) A further observation made when encoding the HOA representation of a typical sound scene is that the number of active predictions, NumActivePred, is often very small. In such a situation, q Instead of using the bit array ActivePred to indicate whether prediction is performed for M, it may be more efficient to transmit or forward the number of active predictions and their respective indices. In particular, this variant of encoding the active ones works for all cases where NumActivePred≦M M It is more efficient if, where M M is the largest integer that satisfies the following equation:

[0062]

number

[0063] In the formula (25),

number

number

[0064] According to the above explanation, a single bit KindOfCodedPredIds can be used to indicate how the index of the direction in which the prediction is to be performed is coded. If the bit KindOfCodedPredIds has the value "1" (or in the alternative "0"), the number NumActivePred and the array PredIds containing the index of the direction in which the prediction is to be performed are coded in the coded side information ζ COD Otherwise, if the bit KindOfCodedPredIds has the value "0" (or alternatively "1"), then the array ActivePred is used to encode the same information. On average, this operation is performed by COD The bit rate for transmission of the signal is reduced over time.

[0065] C) To further increase the side information coding efficiency, the fact that the actually available number of active directional signals used for prediction is often smaller than D is exploited. This is because, for the coding of each element of the index array PredDirSigIds,

number

number

number

number

number

number

number

[0066] The above modifications A) to C) on the known side information encoding process result in the exemplary encoding process depicted in FIG.

[0067] As a result, the coded side information consists of the following components:

number

[0068] The coded representation for the example of equations (7) to (9) is as follows:

[0069]

number

[0070] Advantageously, compared to the state-of-the-art coded representations in equations (20) through (23), this coded representation according to the present invention requires 8 fewer bits.

[0071] It is also possible for the encoder not to provide the bit array PredType.

[0072] Decoding modified side information coding related to spatial prediction The decoding of the modified side information related to spatial prediction is summarized in the exemplary decoding process depicted in Figures 7 and 8 (the process depicted in Figure 8 is a continuation of the process depicted in Figure 7) and described below.

[0073] First, the vector p TYPE and matrix P IND and P Q,F All elements of are initialized by 0. Then the bit PSPredictionActive is read, which indicates whether spatial prediction is performed at all. In case of spatial prediction (i.e. PSPredictionActive=1), the bit KindOfCodedPredIds is read, which indicates the kind of coding of the index of the direction in which the prediction should be performed.

[0074] If KindOfCodedPredIds=0, the bit array ActivePred of length O is read. The q-th element of this array has the direction Ω qIn the next step, the number of predictions NumActivePred is calculated from the array ActivePred, and a bit array PredType of length NumActivePred is read. The elements of this array indicate the type of prediction to be performed for each associated direction. Using the information contained in ActivePred and PredType, a vector p TYPE The elements of are calculated.

[0075] The encoder does not provide the bit array PredType, but instead uses the bit array ActivePred to generate the vector p TYPE It is also possible to calculate the elements of

[0076] If KindOfCodedPredIds=1,

number

number

[0077] The encoder does not provide the bit array PredType, but instead calculates the vector p from the number NumActivePred and the data array PredIds. TYPE It is also possible to calculate the elements of

[0078] In both cases (i.e., KindOfCodedPredIds=0 and KindOfCodedPredIds=1), the next step is to determine NumActivePred·D PRED The array PredDirSigIds is read, which consists of elements. Each element is

number

[0079]

number

[0080] Finally, B SC The array QuantPredGains, which consists of NumNonZeroIds elements encoded by bits, is read. IND Using the information contained in QuantPredGains, we can calculate the matrix P Q,F The elements are set.

[0081] The processing of the invention can be carried out by a single processor or electronic circuitry, or by several processors or electronic circuits operating in parallel and / or acting on different parts of the processing of the invention.

[0082] Several aspects will be described. [Aspect 1] 1. A method for improving the coding of side information required to code a Higher Order Ambisonics representation (HOA) of a sound field having an input time frame of a sequence of HOA coefficients, comprising: determining a dominant directional signal and a residual ambient HOA component; and using a prediction on said dominant directional signal, thereby providing, for a coded frame of HOA coefficients, side information data (ζ(k-2)) describing said prediction, said side information data (ζ(k-2)) comprising: · A bit array indicating whether prediction is performed for a direction (ActivePred); A data array (PredDirSigIds) whose elements represent the indices of the directional signals to be used for the prediction to be performed; a data array (QuantPredGains) whose elements represent quantized scaling factors; The method involves: providing a bit value (PSPredictionActive) indicating whether the prediction should be performed (19;34,384); omitting the bit array and the data array in the side information data (ζ(k−2)) if no prediction is to be performed; If the prediction should be performed, instead of the bit array (ActivePred) indicating whether a prediction is performed for a direction or not, a bit value (KindOfCodedPredIds) is provided indicating whether the number of active predictions (NumActivePred) and a data array (PredIds) containing indices of the directions in which the prediction should be performed are included in the side information data (ζ(k-2)). A method comprising the steps of: [Aspect 2] 1. An apparatus for improving the coding of side information required for coding a Higher Order Ambisonics representation (HOA) of a sound field having an input time frame of a sequence of HOA coefficients, the apparatus comprising: a dominant directional signal and a residual ambient HOA component are determined, and a prediction is used for said dominant directional signal, thereby providing, for a coded frame of HOA coefficients, side information data (ζ(k-2)) describing said prediction, said side information data (ζ(k-2)) comprising: · A bit array indicating whether prediction is performed for a direction (ActivePred); A data array (PredDirSigIds) whose elements represent the indices of the directional signals to be used for the prediction to be performed; a data array (QuantPredGains) whose elements represent quantized scaling factors; The device: providing a bit value (PSPredictionActive) indicating whether the prediction should be performed; omitting the bit array and the data array in the side information data (ζ(k−2)) if no prediction is to be performed; If the prediction should be performed, instead of the bit array (ActivePred) indicating whether a prediction is performed for a direction or not, a bit value (KindOfCodedPredIds) is provided indicating whether the number of active predictions (NumActivePred) and a data array (PredIds) containing indices of the directions in which the prediction should be performed are included in the side information data (ζ(k-2)). An apparatus comprising the means (19;34,384). [Aspect 3] During the encoding of the HOA representation, an estimation of the dominant sound source direction (13) is performed, and a data set of indices of detected directional signals is generated.

number

number

number

number

number

number

number

number

number

Claims

1. 1. A method for decoding a bitstream including an encoded Higher Order Ambisonics (HOA) representation, the method comprising: reading the bits KindOfCodedPredIds; determining an array ActivePred based on the determination that KindOfCodedPredIds=0, each element of the first array ActivePred indicating whether prediction is performed for a corresponding direction; type determining the vector p type is determined based on the sequence ActivePred; The vector p type A matrix P represents an index for which a prediction for a direction is performed from the corresponding directional signal based on IND and determining method.

2. A non-transitory storage medium containing, storing or recording the digital audio signal of claim 1.

3. A non-transitory computer readable medium storing a computer program which, when executed by a processor, performs the method of claim 1.

4. 1. An apparatus for decoding a bitstream including an encoded Higher Order Ambisonics (HOA) representation, the apparatus comprising: a first processor reading bits KindOfCodedPredIds; determining an array ActivePred based on the determination that KindOfCodedPredIds=0, each element of the array ActivePred indicating whether prediction is performed for a corresponding direction; type determining the vector p type is determined based on the array ActivePred; and The vector p type A matrix P represents an index for which a prediction for a direction is performed from the corresponding directional signal based on IND and a third processor that determines Device.

Images