Apparatus and method for encoding or decoding AR / VR metadata using generic codebook

Abstract

An apparatus (100) for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is provided. The apparatus (100) comprises at least one entropy decoding module (110) for decoding the encoded additional audio information when the encoded additional audio information is entropy encoded to obtain decoded additional audio information. Furthermore, the apparatus (100) comprises a signal processor (120) for generating the one or more audio output signals from the one or more encoded audio signals and from the decoded additional audio information.

Description

[0001] manual

[0002] The present invention relates to a device and method for encoding or decoding, and in particular, to a device and method for encoding or decoding augmented reality (AR) or virtual reality (VR) metadata with a universal codebook.

[0003] Further improvement and development of audio coding technology is an ongoing task in audio coding research, wherein the aim is, for example, to create a realistic audio experience for the listener in augmented reality or virtual reality scenarios taking into account audio effects (such as reverberations caused by objects, reflections on walls), while at the same time aiming to encode and decode the audio information with high efficiency.

[0004] One of these new audio technologies that aims to create an improved listening experience for augmented reality or virtual reality is, for example, MPEG-I. MPEG-I is a new developing standard for virtual and augmented reality applications. Its goal is to create a natural, realistic AR or VR experience and deliver an overall convincing experience not only to the eyeballs but also to the ears.

[0005] For example, using MPEG-I technology, when listening to a concert in VR, the listener is no longer fixed to just one point, but can move freely around the concert hall. Or, for example, MPEG-I technology can be used for broadcasting of e-sports or sports events, where users can move around the stadium while watching the game.

[0006] Previous solutions enabled visual or auditory experiences from one observation point with so-called three degrees of freedom (3DoF). In contrast, the upcoming MPEG-I standard supports full six degrees of freedom (6DoF). With 3DoF, the user can move his head freely and receive input from multiple sides. But with 6DoF, the user can move within the virtual space. The user can walk around, explore every perspective, and even interact with the virtual world. MPEG-I technology is also suitable for augmented reality (AR), in which users act within the real world that has been extended by virtual elements. For example, you can arrange several virtual musicians in your living room and enjoy your own personal concert.

[0007] To achieve this goal, MPEG-I provides complex techniques to generate a convincing and highly immersive audio experience, and involves considering many aspects of acoustics. One example is the propagation of sound waves in a room and around obstacles. Another example is a sound source, which can be static or in motion, where a sound source in motion generates a Doppler effect. Sound propagation should have realistic radiation patterns and sizes. For example, MPEG-I technology aims to take into account the diffraction of sound around obstacles or in the corners of a room, and aims to provide efficient rendering of these effects.

[0008] Overall, MPEG-I aims to provide a long-term stable format for rich VR and AR content. Reproduction using MPEG-I is possible with dedicated receiver devices as well as everyday smartphones. MPEG-I aims to distribute VR and AR content as a next-generation video service via existing distribution channels, allowing providers to offer users a truly exciting and immersive experience with entertainment, documentary, educational or sports content.

[0009] There is a need to provide the decoder with additional audio information, for example information about the real or virtual acoustic environment and / or its effects (such as reverberation). Providing such information in an efficient manner would be highly appreciated.

[0010] In summary, it would be highly appreciated if improved concepts for audio encoding and audio decoding could be provided.

[0011] It is an object of the invention to provide an improved concept for audio encoding and audio decoding. The objects of the invention are solved by the subject matter of the independent claims. Specific embodiments are provided in the dependent claims.

[0012] A device for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is provided. The device comprises at least one entropy decoding module, the at least one entropy decoding module being used to decode the encoded additional audio information when the encoded additional audio information is entropy encoded to obtain decoded additional audio information. In addition, the device comprises a signal processor, the signal processor being used to generate one or more audio output signals according to the one or more encoded audio signals and according to the decoded additional audio information.

[0013] In addition, a device for encoding one or more audio signals and additional audio information according to an embodiment is provided. The device comprises an audio signal encoder for encoding the one or more audio signals to obtain one or more encoded audio signals. In addition, the device comprises at least one entropy coding module, and the at least one entropy coding module is used to encode the additional audio information using entropy coding to obtain the encoded additional audio information.

[0014] In addition, a device for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is provided. The device comprises an input interface for receiving one or more encoded audio signals and for receiving additional audio information data. In addition, the device comprises a signal generator, and the signal generator is used to generate one or more audio output signals according to the encoded audio signal and according to the second additional audio information. The signal generator is configured to use the additional audio information data and use the first additional audio information to obtain the second additional audio information when the additional audio information data exhibits a redundant state. In addition, the signal generator is configured to use the additional audio information data without using the first additional audio information to obtain the second additional audio information when the additional audio information data exhibits a non-redundant state.

[0015] In addition, a device for encoding one or more audio signals and for generating additional audio information data according to an embodiment is provided. The device includes an audio signal encoder for encoding one or more audio signals to obtain one or more encoded audio signals. In addition, the device includes an additional audio information generator for generating additional audio information data, wherein the additional audio information generator exhibits a non-redundant operating mode and a redundant operating mode. The additional audio information generator is configured to generate additional audio information data when the additional audio information generator exhibits a non-redundant operating mode, so that the additional audio information data includes the second additional audio information. In addition, the additional audio information generator is configured to generate additional audio information data when the additional audio information generator exhibits a non-redundant operating mode, so that the additional audio information data does not include the second additional audio information or only includes a part of the second additional audio information, so that the additional audio information data can be used together with the first additional audio information to obtain the second additional audio information.

[0016] Furthermore, a method for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is provided. The method comprises:

[0017] - when the encoded additional audio information is entropy encoded, decoding the encoded additional audio information to obtain decoded additional audio information. And:

[0018] - generating one or more audio output signals from the one or more encoded audio signals and from the decoded additional audio information.

[0019] Furthermore, a method for encoding one or more audio signals and additional audio information according to an embodiment is provided. The method comprises:

[0020] - encoding one or more audio signals into one or more encoded audio signals. And:

[0021] - encoding the additional audio information using entropy coding to obtain encoded additional audio information.

[0022] Furthermore, a method for generating one or more audio output signals from one or more encoded audio signals according to another embodiment is provided. The method comprises:

[0023] - receiving one or more encoded audio signals and for receiving additional audio information data. And:

[0024] - generating one or more audio output signals depending on the encoded audio signal and depending on the second additional audio information.

[0025] The method comprises: if the additional audio information data exhibits a redundant state, using the additional audio information data and using the first additional audio information to obtain the second additional audio information. In addition, the method comprises: if the additional audio information data exhibits a non-redundant state, using the additional audio information data without using the first additional audio information to obtain the second additional audio information.

[0026] Furthermore, a method according to an embodiment for encoding one or more audio signals and for generating additional audio information data is provided.

[0027] The method includes:

[0028] - encoding one or more audio signals to obtain one or more encoded audio signals. And:

[0029] -Generate additional audio information data.

[0030] In a non-redundant operating mode, generating the additional audio information data is performed such that the additional audio information data includes the second additional audio information. In a redundant operating mode, generating the additional audio information data is performed such that the additional audio information data does not include the second additional audio information or includes only a portion of the second additional audio information, so that the second additional audio information can be obtained using the additional audio information data together with the first additional audio information.

[0031] Furthermore, computer programs are provided, wherein each of the computer programs is configured to implement one of the above methods when executed on a computer or a signal processor.

[0032] In the following, embodiments of the present invention are described in more detail with reference to the accompanying drawings, in which:

[0033] Figure 1 A device for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is presented.

[0034] Figure 2A device for generating one or more audio output signals according to another embodiment is presented, the device further comprising at least one non-entropy decoding module and a selector.

[0035] Figure 3 A device for generating one or more audio output signals according to another embodiment is shown, wherein the device comprises a non-entropy decoding module, a Huffman decoding module and an arithmetic decoding module.

[0036] Figure 4 An apparatus for encoding one or more audio signals and additional audio information according to an embodiment is presented.

[0037] Figure 5 A device for encoding one or more audio signals and additional audio information according to a further embodiment is shown, the device comprising at least one non-entropy coding module and a selector.

[0038] Figure 6 A device for generating one or more audio output signals according to another embodiment is shown, wherein the device comprises a non-entropy coding module, a Huffman coding module and an arithmetic coding module.

[0039] Figure 7 A system according to an embodiment is presented.

[0040] Figure 8 Specific embodiments are shown that depict encoding of additional audio data and decoding of the encoded additional audio data.

[0041] Fig. 9 A device for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is presented.

[0042] Fig.10 A device for encoding one or more audio signals and for generating additional audio information data according to an embodiment is presented.

[0043] Fig.11 A system according to another embodiment is presented.

[0044] Figure 1 A device 100 for generating one or more audio output signals from one or more encoded audio signals according to an embodiment is shown.

[0045] The apparatus 100 comprises at least one entropy decoding module 110 for decoding the encoded additional audio information when the encoded additional audio information is entropy encoded to obtain decoded additional audio information.

[0046] Furthermore, the device 100 comprises a signal processor 120 for generating one or more audio output signals from the one or more encoded audio signals and from the decoded additional audio information.

[0047] Figure 2 A device 100 for generating one or more audio output signals according to another embodiment is shown, wherein, compared to Figure 1 The device 100, Figure 2 The device 100 further includes at least one non-entropy decoding module 111 and a selector 115.

[0048] The at least one non-entropy decoding module 111 may, for example, be configured to decode the encoded additional audio information when the encoded additional audio information has not been entropy encoded to obtain decoded additional audio information.

[0049] The selector 115 may, for example, be configured to select one of the at least one entropy decoding module 110 and the at least one non-entropy decoding module 111 for decoding the encoded additional audio information according to whether the encoded additional audio information is entropy encoded.

[0050] According to an embodiment, the encoded additional audio information may, for example, contain augmented reality data or virtual reality data.

[0051] In an embodiment, the encoded additional audio information depends on the real listening environment or on a virtual listening environment or on an enhanced listening environment.

[0052] In a typical application scenario, the listening environment should be modeled and encoded on the encoder side, and the modeling of the listening environment should be received on the decoder side.

[0053] Typical additional audio information relevant to the listening environment may be, for example, information about a number of reflecting objects, where sound waves may be, for example, reflected at these objects. Generally speaking, reflecting objects relevant for reflection are those reflecting objects that have an extension that is (significantly) larger than the wavelength of the audible sound. Therefore, walls or other large reflecting objects are particularly important when considering reflections. These reflecting objects may, for example, be suitably represented by surfaces, at which the sound is reflected.

[0054] In a three-dimensional environment, a surface may be characterized, for example, by three points in a three-dimensional coordinate system, wherein each of the three points may be defined, for example, by its x-coordinate value, its y-coordinate value, and its z-coordinate value. Thus, for each of the three points, three x, y, z values ​​are required, and thus a total of nine coordinate values ​​are required to define the surface.

[0055] This can be done, for example, by using the surface normal vector A more efficient representation of the surface is achieved by defining the surface using a scalar distance value d that defines the distance from the defined origin to the surface. Defined by azimuth and elevation (the normal vector has length 1 and therefore does not need to be encoded), the surface can thus be represented by only three values, namely the scalar distance value d to the surface, and the normal vector to the surface It is defined by the azimuth and elevation angles.

[0056] Typically, for efficient coding, the azimuth and elevation angles may be appropriately quantized, for example. For example, each azimuth angle may have 2 n different azimuth values, and the elevation angles may, for example, be encoded so that each elevation angle may have 2 n-1 one of 4 different elevation values.

[0057] As outlined above, the representation of walls plays an important role when defining a listening environment focused on reflections. This is true for indoor scenarios, where walls play a very important role for, e.g., early reflections. However, this is also true for outdoor scenarios, where the walls of buildings represent the main part of the relevant reflections.

[0058] It is observed that in common environments, many walls are at approximately 90° angles to each other. For example, in an indoor situation, there are many horizontal walls and vertical walls. Although it has been found that due to construction deviations, the relationship between walls is not always exactly 90°, but may be, for example, 89.8°, 89.6°, 90.3° or the like, there is still a significant proportion of walls that have a relationship of about 90° and about 0° relative to each other.

[0059] For example, if the wall is a horizontal wall, the elevation angle of the wall may be defined as 0°, for example, and if the surface of the wall is a vertical wall, it may be defined as 90°, for example. Then, in a real-world example, there will be a significant proportion of walls with elevation angles of about 90° (e.g., 89.8°, 89.7°, 90.2°) and a significant proportion of walls with elevation angles of about 0° (e.g., 0.3°, -0.2°, 0.4°).

[0060] The same observations made for elevation generally apply to azimuth, since rooms generally have a rectangular shape.

[0061] Returning to the example of elevation, however, it should be noted that if the 0° value of elevation is defined differently than above, common walls exhibit other values. For example, if a surface inclined 20° relative to the horizontal is defined as having a 0° elevation, many real-world walls may, for example, have an elevation of about -20° (e.g., -19.8°, -20.0°, -20.2°), and many real-world walls may, for example, have an elevation of about 70° (e.g., 69.8°, 70.0°, 70.2°). Nevertheless, a significant proportion of walls will have the same elevation at certain elevations (about -20° and about 70° in this example). The same applies to azimuth.

[0062] Furthermore, some other walls will have some other typical elevation angles. For example, roofs are often tilted 45° or 35° or 30°. These values ​​also appear with some frequency in real-world examples.

[0063] Furthermore, it should be noted that not all real-world rooms have a rectangular ground plane shape, but may, for example, exhibit other regular shapes. For example, consider a room with an octagonal ground plane shape. Although present, it can be assumed that some azimuth angles, such as azimuth angles of approximately 0°, 45°, 90°, and 135°, occur more frequently than other azimuth angles.

[0064] Furthermore, in outdoor examples, walls will often exhibit similar azimuths. For example, two parallel walls of a house will exhibit similar azimuths, but this may also be the case, for example, with walls of adjacent houses, which are typically built in rows with regular, similar ground shapes relative to each other. Walls of adjacent houses will also exhibit similar azimuth values ​​and therefore have similarly oriented reflective walls / surfaces.

[0065] Based on the above observations, it has been found that it is often particularly suitable to use entropy coding to encode and decode the additional audio information. This applies in particular to situations where certain values ​​among all possible values ​​occur (significantly) more frequently than other values.

[0066] In a particular embodiment, elevation values ​​of a surface (eg, representing a reflective object) may be encoded and decoded, for example, using entropy coding (eg, using Huffman coding or using arithmetic coding).

[0067] Likewise, in certain embodiments, azimuth values ​​of a surface (eg, representing a reflective object) may be encoded and decoded, for example, using entropy coding (eg, using Huffman coding or using arithmetic coding).

[0068] The above considerations also apply to other application scenarios. For example, for a given audio source position s, and for example for a given listener position l, the reflection sequence may, for example, define some one or more surfaces identified by some one or more surface indices, wherein the one or more surface indices define the surfaces on which a sound wave from the audio source is reflected on a certain propagation path until it reaches the (audible) listener position.

[0069] For example, for a source at position s and a listener at position l, the reflection sequence [5, 18] defines that on a specific propagation path, the sound wave from the source at position s is first reflected at the surface of surface index 5, and then reflected at the surface of surface index 18, until it finally reaches the listener at position l (audibly, so that the listener can still perceive the sound wave). The second reflection sequence may, for example, be the reflection sequence [3, 12]. The third reflection sequence only includes [5], indicating that on a specific propagation path, the sound wave from the sound source s is reflected only by surface 5 and then reaches position l where the listener can hear it. The fourth reflection sequence [3, 7] defines that on a specific propagation path, the sound wave from the source s is first reflected at the surface of surface index 3, and then reflected at the surface of surface index 7, until it finally reaches the listener audibly. All reflection sequences for the listener at position l and for the source at position s together define a set of reflection sequences for the listener at position l and for the source at position s.

[0070] However, there may also be, for example, other defined surfaces, such as surfaces with surface index 6, 8, 9, 10, 11 or 15, which may, for example, be located at position l away from the listener and at position s away from the source. For a listener at position l and for a source at position s, these surfaces will appear less frequently or not at all in the set of reflection sequences. From this observation it has been found that, in general, it is desirable to use entropy coding to encode the set of reflection sequences.

[0071] Furthermore, even if multiple sets of reflection sequences are jointly encoded for multiple different listener positions and / or multiple different source positions, it is still desirable to employ entropy coding. For example, in certain listening environments, areas reachable by a user may be defined, wherein it may be assumed, for example, that the user never moves through dense bushes or other inaccessible areas. In some application scenarios, reflection sequence sets for user positions in these inaccessible areas are not provided. Therefore, walls in these areas are typically less present in multiple sets of reflection sequences, since these walls are located far from all defined possible user positions. This causes different appearances of surface indices in multiple reflection sequence sets, and therefore, it is proposed to entropy code these surface indices in the reflection sets.

[0072] In one embodiment, the actual occurrence of different values ​​of the additional audio information may be observed, for example, and based on this observation, entropy coding or non-entropy coding may be employed, for example. Using non-entropy coding is particularly advantageous when the occurrence of different values ​​is rendered with the same or at least approximately similar frequency: for example, predefined codewords and symbol relationships may be employed, which do not have to be transmitted from the encoder to the decoder.

[0073] Returning again to the more general examples, these examples can also be applied to other application examples than the one just described:

[0074] According to an embodiment, the encoded additional audio information may, for example, include propagation information according to one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment.

[0075] In an embodiment, the propagation information may be, for example, reflection information based on one or more reflections of one or more sound waves propagating along one or more propagation paths at one or more reflection objects in a real listening environment, in a virtual listening environment, or in an enhanced listening environment.

[0076] According to an embodiment, the propagation information may, for example, be diffraction information based on one or more diffractions of one or more sound waves propagating along one or more propagation paths at one or more diffraction objects in a real listening environment, in a virtual listening environment, or in an enhanced listening environment.

[0077] According to an embodiment, the encoded additional audio information may, for example, comprise data for rendering early reflections.The signal processor 120 may, for example, be configured to generate one or more audio output signals depending on the data for rendering early reflections.

[0078] In an embodiment, the signal processor 120 may, for example, be configured to generate a binaural signal comprising two binaural channels as the one or more audio output signals.

[0079] According to an embodiment, the at least one entropy decoding module 110 may, for example, include a Huffman decoding module 116 for decoding the encoded additional audio information when the encoded additional audio information is Huffman encoded.

[0080] In an embodiment, the at least one entropy decoding module 110 may, for example, include an arithmetic decoding module 118 for decoding the encoded additional audio information when the encoded additional audio information is arithmetically encoded.

[0081] Figure 3An apparatus 100 for generating one or more audio output signals according to another embodiment is shown, wherein the apparatus 100 comprises a non-entropy decoding module 111 , a Huffman decoding module 116 and an arithmetic decoding module 118 .

[0082] The selector 115 may, for example, be configured to select at least one of the non-entropy decoding module 111 and the Huffman decoding module 116 and the arithmetic decoding module 118 for decoding the encoded additional audio information.

[0083] According to an embodiment, the at least one non-entropy decoding module 111 may, for example, include a fixed length decoding module for decoding the encoded additional audio information when the encoded additional audio information is fixed length encoded.

[0084] In an embodiment, the apparatus 100 may, for example, be configured to receive selection information. The selector 115 may, for example, be configured to select one of the at least one entropy decoding module 110 and the at least one non-entropy decoding module 111 according to the selection information.

[0085] According to an embodiment, the device 100 may, for example, be configured to receive a codebook or a coding tree, on which the encoded additional audio information depends.At least the entropy decoding module 110 may, for example, be configured to decode the encoded additional audio information using the codebook or using the coding tree.

[0086] In an embodiment, the device 100 may be configured to receive an encoding of a structure of a coding tree, and the encoded additional audio information depends on the coding tree. At least the entropy decoding module 110 may be configured to reconstruct a plurality of codewords of the coding tree according to the structure of the coding tree. In addition, at least the entropy decoding module 110 may be configured to decode the encoded additional audio information using the codewords of the coding tree.

[0087] For example, typical encoding information that can be transmitted, for example, from an encoder to a decoder can be, for example, a codeword list of N elements containing all N codewords of a code and a symbol list containing all N symbols encoded by the N codewords of the code. The encoding information can be defined as follows: the codeword at position p (where 1≤p≤N) of the codeword list encodes the symbol at position p of the symbol list.

[0088] For example, the contents of the following two lists may be transmitted, where each symbol may represent, for example, a surface index identifying a specific surface:

[0089] Codeword 00 01 10 110 1110 1111 symbol 18 23 15 3 7 9

[0090] However, according to an embodiment, instead of transmitting the list of codewords, a representation of the coding tree may be transmitted, for example from the encoder, and may be received, for example by the decoder. The decoder may, for example, be configured to construct the list of codewords from the received representation of the coding tree.

[0091] For example, each internal node (eg, except the root node of the coding tree) may be represented, for example, by a first bit value (eg, 0), and each leaf node of the coding tree may be represented, for example, by a second bit value (eg, 1).

[0092] Considering the above list of codewords,

[0093] Codeword 00 01 10 110 1110 1111

[0094] The coding tree is traversed from the leftmost branch to the rightmost branch, all new internal nodes are encoded with 0 when traversing the coding tree, and all leaf nodes are encoded with 1 when traversing the coding tree, thereby generating the encoding of the coding tree with the above codeword, which is expressed as:

[0095] Use codeword to reach leaf node 00 01 10 110 1110 1111 Bit 01 1 01 01 01 1

[0096] The resulting coding tree is represented as: 01 1 01 01 01 1.

[0097] On the decoder side, the representation of the coding tree can be parsed into a list of codewords:

[0098] Codeword 1: The first leaf node appears at the second node: codeword 1, bit 00.

[0099] Codeword 2: Next is another leaf node: codeword 2, bit 01.

[0100] Codeword 3: All nodes to the left of the root node have been found, continuing the right branch of the root node: the first leaf on the right side of the root node is at the second node: Codeword 3, bit "10".

[0101] Code word 4: Move up one node (under the first branch 1). Descend to the right branch (second branch 1), internal node (0); move to the left branch (branch 0), leaf node (1): Code word 4: "110". (Leaf node under branch 1-1-0)

[0102] Code word 5: Move up one node (under the second branch 1). Descend to the right branch (third branch 1), internal node (0); move to the left branch (branch 0), leaf node (1): Code word 5: "1110" (leaf node under branch 1-1-1-0)

[0103] Code word 6: Promote one node up and descend to the right branch (the fourth branch 1), which is the leaf node (1): Code word 6: "1111" (leaf node under branch 1-1-1-1).

[0104] By encoding the coding tree structure instead of the codewords, the coding efficiency is increased.

[0105] In an embodiment, the apparatus 100 may, for example, further include a memory having a codebook or a coding tree stored thereon.At least the entropy decoding module 110 may, for example, be configured to decode the encoded additional audio information using a codebook or using a coding tree.

[0106] According to an embodiment, the apparatus 100 may, for example, be configured to receive encoded additional audio information comprising a plurality of transmitted symbols and an offset value. The at least one non-entropy decoding module 111 may, for example, be configured to decode the encoded additional audio information using the plurality of transmitted symbols and using the offset value.

[0107] In an embodiment, the data used to render early reflections may, for example, contain information about the position of one or more walls, which are one or more real walls or virtual walls in the environment. The signal processor 120 may, for example, be configured to generate one or more audio output signals based on the information about the position of one or more walls.

[0108] According to an embodiment, the information about each of the one or more walls may, for example, include information about the azimuth and / or elevation of the wall, wherein the azimuth of the wall may, for example, be entropy encoded and / or the elevation of the wall may, for example, be entropy encoded. One or more entropy decoding modules of the at least one entropy decoding module 110 are configured to decode the entropy encoded azimuth of the wall and / or the entropy encoded elevation of the wall.

[0109] In an embodiment, one or more of the at least one entropy decoding module 110 is configured to decode the entropy encoded azimuth angle of the wall and / or the entropy encoded elevation angle of the wall using a codebook or a coding tree.

[0110] According to an embodiment, the encoded additional audio information may, for example, include voxel position information, wherein the position information may, for example, include information about one or more positions of one or more voxels of a plurality of voxels within a three-dimensional coordinate system. The signal processor 120 may, for example, be configured to generate one or more audio output signals based on the voxel position information.

[0111] In an embodiment, the at least one entropy decoding module 110 may, for example, be configured to decode the entropy-encoded encoded additional audio information, wherein the entropy-encoded encoded additional audio information may, for example, include at least one of the following:

[0112] - List of triangle indices, such as earlySurfaceFaceIdx,

[0113] - The array length of the list of triangle indices, such as the array length of earlySurfaceFaceIdx, such as earlySurfaceLengthFaceIdx,

[0114] - an array with azimuths specifying the surface normal in spherical coordinates (e.g. in the form of Hesse normals), e.g. earlySurfaceAzi,

[0115] - an array with elevation angles, specifying the surface normal in spherical coordinates (e.g. in Hesse normal form), e.g. earlySurfaceEle,

[0116] - an array with distance values ​​(e.g. in Hesse normal form), e.g. earlySurfaceDist,

[0117] - an array with listener positions, e.g. an array with listener voxel indices, e.g. earlyVoxelL,

[0118] - an array with one or more sound source positions, e.g. an array with source voxel indices, e.g. earlyVoxelS,

[0119] - a removal list or removal set, such as a differently encoded removal list or a differently encoded removal set, which specifies the reflection sequence indices of a set of reflection sequences that should be removed or a list of reference reflection sequences that should be removed, such as earlyVoxelIndicesRemovedDiff,

[0120] - Several reflection sequences or several reflection paths, such as earlyVoxelNumPaths

[0121] - an array, e.g. a 2D array, specifying the reflection order, e.g. earlyVoxelOrder

[0122] -Reflection sequences, such as earlyVoxelSurf.

[0123] Figure 4 An apparatus 200 for encoding one or more audio signals and additional audio information according to an embodiment is shown.

[0124] The apparatus 200 comprises an audio signal encoder 210 for encoding one or more audio signals to obtain one or more encoded audio signals.

[0125] Furthermore, the apparatus 200 comprises at least one entropy coding module 220 for encoding the additional audio information using entropy coding to obtain encoded additional audio information.

[0126] Figure 5 A device 200 for encoding one or more audio signals and additional audio information according to another embodiment is shown. Figure 4 Device 200, Figure 4The device 200 further includes at least one non-entropy coding module 221 and a selector 215.

[0127] The at least one non-entropy encoding module 221 may, for example, be configured to encode the additional audio information to obtain encoded additional audio information, and

[0128] The selector 215 may, for example, be configured to select one of the at least one entropy coding module 220 and the at least one non-entropy coding module 221 for encoding the additional audio information according to a distribution of symbols within the additional audio information to be encoded.

[0129] According to an embodiment, the encoded additional audio information may, for example, contain augmented reality data or virtual reality data.

[0130] In an embodiment, the encoded additional audio information depends on the real listening environment or on a virtual listening environment or on an enhanced listening environment.

[0131] According to an embodiment, the additional audio information may, for example, include propagation information according to one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment.

[0132] In an embodiment, the propagation information may be, for example, reflection information based on one or more reflections of one or more sound waves propagating along one or more propagation paths at one or more reflection objects in a real listening environment, a virtual listening environment, or an enhanced listening environment.

[0133] According to one embodiment, the propagation information may, for example, be diffraction information based on one or more diffraction of one or more sound waves propagating along one or more propagation paths at one or more diffraction objects in a real listening environment, in a virtual listening environment, or in an enhanced listening environment.

[0134] According to an embodiment, the encoded additional audio information may, for example, contain data for rendering early reflections.

[0135] In an embodiment, the at least one entropy encoding module 220 may, for example, include a Huffman encoding module 226 for encoding the additional audio information using Huffman coding.

[0136] According to an embodiment, the at least one entropy coding module 220 may, for example, include an arithmetic coding module 228 for encoding the additional audio information using arithmetic coding.

[0137] Figure 6An apparatus 200 for generating one or more audio output signals according to another embodiment is shown, wherein the apparatus 200 comprises a non-entropy coding module 221 , a Huffman coding module 226 and an arithmetic coding module 228 .

[0138] The selector 215 may, for example, be configured to select at least one non-entropy coding module 221 and one of the Huffman coding module 226 and the arithmetic coding module 228 for encoding the additional audio information.

[0139] In an embodiment, the at least one non-entropy coding module 221 may, for example, comprise a fixed length coding module for encoding the additional audio information.

[0140] According to an embodiment, the apparatus 200 may, for example, be configured to generate selection information indicating one of the at least one entropy encoding module 220 and the at least one non-entropy encoding module 221 that have been used to encode the additional audio information.

[0141] In an embodiment, the apparatus 200 may, for example, be configured to transmit a codebook or a coding tree that has been used to encode the additional audio information.

[0142] In an embodiment, the device 200 may, for example, be configured to transmit an encoding of the structure of a coding tree, the encoded additional audio information depending on the coding tree.

[0143] According to an embodiment, the apparatus 200 may, for example, further include a memory having a codebook or a coding tree stored thereon.At least the entropy encoding module 220 may, for example, be configured to encode the additional audio information using a codebook or using a coding tree.

[0144] In an embodiment, the at least one entropy encoding module 220 may, for example, be configured to encode the additional audio information such that the encoded additional audio information may, for example, comprise a plurality of transmitted symbols and an offset value.

[0145] According to an embodiment, the data used for rendering early reflections may, for example, contain information about the position of one or more walls, being one or more real walls or virtual walls in the environment.

[0146] In an embodiment, the information about each of the one or more walls may, for example, include information about the azimuth and / or elevation of the wall, wherein the azimuth of the wall may, for example, be entropy encoded and / or the elevation of the wall may, for example, be entropy encoded. One or more entropy encoding modules of the at least one entropy encoding module 220 are configured to encode additional audio information, such that the encoded additional audio information may, for example, include the entropy encoded azimuth of the wall and / or the entropy encoded elevation of the wall.

[0147] According to an embodiment, the one or more entropy coding modules are configured to encode the entropy coded azimuth angle of the wall and / or the entropy coded elevation angle of the wall using a codebook or a coding tree.

[0148] In an embodiment, the encoded additional audio information may, for example, include voxel position information, wherein the position information may, for example, include information about one or more positions of one or more voxels of the plurality of voxels within a three-dimensional coordinate system.

[0149] According to an embodiment, the at least one entropy coding module 220 may, for example, be configured to encode the additional audio information using entropy coding, wherein the encoded additional audio information may, for example, include at least one of the following:

[0150] - List of triangle indices, such as earlySurfaceFaceIdx,

[0151] - The array length of the list of triangle indices, such as the array length of earlySurfaceFaceIdx, such as earlySurfaceLengthFaceIdx,

[0152] - an array with azimuths specifying the surface normal in spherical coordinates (e.g. in the form of Hesse normals), e.g. earlySurfaceAzi,

[0153] - an array with elevation angles, specifying the surface normal in spherical coordinates (e.g. in Hesse normal form), e.g. earlySurfaceEle,

[0154] - an array with distance values ​​(e.g. in Hesse normal form), e.g. earlySurfaceDist,

[0155] - an array with listener positions, e.g. an array with listener voxel indices, e.g. earlyVoxelL,

[0156] - an array with one or more sound source positions, e.g. an array with source voxel indices, e.g. earlyVoxelS,

[0157] - a removal list or removal set, e.g. a removal list encoded in a different way or a removal set encoded in a different way that specifies the reflection sequence indices of a set of reflection sequences that should be removed or a list of reference reflection sequences that should be removed, e.g. earlyVoxelIndicesRemovedDiff,

[0158] - Several reflection sequences or several reflection paths, such as earlyVoxelNumPaths

[0159] - an array, e.g. a 2D array, specifying the reflection order, e.g. earlyVoxelOrder

[0160] -Reflection sequences, such as earlyVoxelSurf.

[0161] Figure 7 A system according to an embodiment is presented. The system comprises Figure 4 The system comprises a device 200 for encoding one or more audio signals and additional audio information to obtain one or more encoded audio signals and encoded additional audio information. Figure 1 The invention relates to a device 100 for generating one or more audio output signals from one or more encoded audio signals depending on encoded additional audio information.

[0162] Figure 8 Specific embodiments depicting encoding of additional audio data and decoding of the encoded additional audio data are shown. Figure 8 In the example, the additional audio data is AR data or VR data, which is encoded on the encoder side to obtain encoded AR data or VR data. Metadata may also be encoded. The encoded AR data or encoded VR data is then decoded on the decoder side to obtain decoded AR data or decoded VR data. On the encoder side, a selector manipulates an encoder switch to select one of N different encoder modules for encoding the AR data or VR data. Figure 8 In the example, the selector provides information to the decoder side so as to select a corresponding decoding module among the N decoding modules for decoding the encoded AR data or the encoded VR data.

[0163] Hereinafter, other embodiments are provided.

[0164] According to an embodiment, a system for encoding and decoding a data sequence is provided, the system having an encoder subsystem and a decoder subsystem. The encoder subsystem may, for example, include at least two different encoding methods, an encoder selector, and an encoder switch for selecting one of the encoding methods. The encoder subsystem may, for example, transmit the selected selection, encoding parameters of the selected encoder, and data encoded by the selected encoder. The decoder subsystem may, for example, include a corresponding decoder and a decoder switch for selecting one of the decoding methods.

[0165] In an embodiment, the data sequence may, for example, include AR / VR data.

[0166] According to an embodiment, the data sequence may, for example, contain metadata for rendering early reflections.

[0167] In an embodiment, at least one fixed length encoder / decoder may, for example, be used, and at least one variable length encoder / decoder may, for example, be used.

[0168] According to an embodiment, one of the variable length encoders / decoders is a Huffman encoder / decoder.

[0169] In an embodiment, the encoding parameters may, for example, include a codebook or a decoding tree.

[0170] According to an embodiment, the encoding parameters may for example comprise an offset value and wherein a combination of the offset value and the transmitted symbols generates a decoded data sequence.

[0171] Fig. 9 A device 300 for generating one or more audio output signals from one or more encoded audio signals according to another embodiment is shown.

[0172] The apparatus 300 comprises an input interface 310 for receiving one or more encoded audio signals and for receiving additional audio information data.

[0173] Furthermore, the device 300 comprises a signal processor 320 for generating one or more audio output signals from the encoded audio signal and from the second additional audio information.

[0174] The signal generator 320 is configured to obtain the second additional audio information using the additional audio information data and using the first additional audio information in case the additional audio information data exhibits a redundant state.

[0175] Furthermore, the signal generator 320 is configured to obtain the second additional audio information using the additional audio information data instead of the first additional audio information if the additional audio information data exhibits a non-redundant state.

[0176] According to an embodiment, the input interface 310 may be configured, for example, to receive propagation information data as additional audio information data. The signal generator 320 may be configured, for example, to generate one or more audio output signals according to the second additional audio information (which is the second propagation information). In addition, the signal generator 320 may be configured, for example, to use the propagation information data and the first additional audio information (which is the first propagation information) to obtain the second propagation information when the propagation information data exhibits a redundant state. In addition, the signal generator 320 may be configured, for example, to use the propagation information data without using the first propagation information to obtain the second propagation information when the propagation information data exhibits a non-redundant state.

[0177] According to an embodiment, the first propagation information and / or the second propagation information may be based on one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment.

[0178] In an embodiment, the propagation information data may include, for example, reflection information data and / or diffraction information data. The first propagation information may include, for example, first reflection information and / or first diffraction information. In addition, the second propagation information may include, for example, second reflection information and / or second diffraction information.

[0179] According to an embodiment, the input interface 310 may be configured to receive the reflection information data as the propagation information data, for example. The signal generator 320 may be configured to generate one or more audio output signals according to the second propagation information (which is the second reflection information), for example. In addition, the signal generator 320 may be configured to use the reflection information data and the first propagation information (which is the first reflection information) to obtain the second reflection information when the reflection information data exhibits a redundant state, for example. In addition, the signal generator 320 may be configured to use the reflection information data and not use the first reflection information to obtain the second reflection information when the reflection information data exhibits a non-redundant state, for example.

[0180] In an embodiment, the first reflection information and / or the second reflection information may, for example, depend on one or more reflections of one or more sound waves propagating along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment at one or more reflecting objects.

[0181] The first reflection information and the second reflection information may, for example, include the set of reflection sequences described above. As previously outlined, for example, for a given audio source position s and, for example, for a given listener position l, the reflection sequence may, for example, define a number of one or more surfaces identified by a number of one or more surface indices, which define the surfaces on which sound waves from the audio source are reflected on a specific propagation path until they reach the (audible) listener position.

[0182] All reflection sequences defined for a listener at position / and for a source at position s form the set of reflection sequences.

[0183] It has been found that, for example, for adjacent listener positions, the sets of reflection sequences are very similar. Therefore, it is proposed that the encoder only encodes reflection sequences (e.g., in the reflection information data) that are not composed of similar reflection sequence sets (e.g., in the first reflection information), and only indicates reflection sequences of similar reflection sequence sets that are invalid for the current reflection sequence set. Similarly, each decoder uses the received reduced information (e.g., the reflection information data) to obtain the current reflection sequence set (e.g., the second reflection information) from the similar reflection sequence set (e.g., the first reflection information).

[0184] In an embodiment, the input interface 310 may, for example, be configured to receive diffraction information data as propagation information data. The signal generator 320 may, for example, be configured to generate one or more audio output signals according to the second propagation information (which is the second diffraction information). In addition, the signal generator 320 may, for example, be configured to use the diffraction information data and the first propagation information (which is the first diffraction information) to obtain the second diffraction information when the diffraction information data exhibits a redundant state. In addition, the signal generator 320 may, for example, be configured to use the diffraction information data without using the first diffraction information to obtain the second diffraction information when the diffraction information data exhibits a non-redundant state.

[0185] According to an embodiment, the first diffraction information and / or the second diffraction information may, for example, depend on one or more diffraction of one or more sound waves propagating along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment at one or more diffraction objects.

[0186] For example, the first diffraction information and the second diffraction information may, for example, comprise a set of diffraction sequences for a listener at position l and a source at position s. The set of diffraction sequences may, for example, be defined similarly to the set of reflection sequences, but with respect to diffraction objects (e.g., objects causing diffraction) rather than reflection objects. Typically, the diffraction object and the reflection object may, for example, be the same object. When these objects are considered as reflection objects, the surfaces of these objects are considered, whereas when these objects are considered as diffraction objects, the edges of these objects are considered for diffraction.

[0187] According to an embodiment, if the propagation information data exhibits a redundant state, the propagation information data may, for example, indicate one or more propagation sequences to be removed from the first propagation information (which is the first set of propagation sequences), and / or may, for example, indicate one or more propagation sequences to be added to the first set of propagation sequences to obtain the second propagation information (which is the second set of propagation sequences). The signal generator 320 may, for example, be configured to update the first set of propagation sequences using the propagation information data to obtain the second set of propagation sequences.

[0188] In an embodiment, each reflection sequence in the first set of reflection sequences and the second set of reflection sequences may, for example, indicate a group of one or more reflection objects or a group of one or more diffraction objects.

[0189] In an embodiment, if the propagated information data exhibits a non-redundant state, the propagated information data may, for example, include a second set of propagation sequences, and the signal generator 320 may, for example, be configured to determine the second set of propagation sequences from the propagated information data.

[0190] According to an embodiment, the first set of propagation sequences may be associated with a first listener position and with a first source position, for example. The second set of propagation sequences may be associated with a second listener position and with a second source position, for example. The first listener position may be different from the second listener position, for example, and / or wherein the first source position may be different from the second source position, for example.

[0191] In an embodiment, the first set of propagation sequences may, for example, be a first set of reflection sequences. The second set of propagation sequences may, for example, be a second set of reflection sequences. Each reflection sequence in the first set of reflection sequences may, for example, contain information about a group of one or more reflection objects of the reflection sequence, where a sound wave emitted by an audio source at a first source position and perceptible by a listener at a first listener position is reflected on the way to a current listener position. Each reflection sequence in the second set of reflection sequences may, for example, contain information about a group of one or more reflection objects of the reflection sequence, where a sound wave emitted by an audio source at a second source position and perceptible by a listener at a second listener position is reflected on the way to the current listener position.

[0192] According to an embodiment, the one or more encoded audio signals are associated with an audio source located at a source position of the second set of reflection sequences. The signal generator 320 may, for example, be configured to generate one or more audio output signals using the one or more encoded audio signals and using the second set of reflection sequences, such that the one or more audio output signals may, for example, contain early reflections of sound waves emitted by the audio source at the source position of the second set of reflection sequences.

[0193] In an embodiment, the input interface 310 may be configured to receive reflection information data as propagation information data, for example. The signal generator 320 may be configured to obtain a plurality of reflection sequence sets, each of which may be associated with a listener position and with a source position, for example. The input interface 310 may be configured to receive an indication, for example. In order to determine the second set of reflection sequences, the signal generator 320 may be configured to use the indication to determine the first listener position and the first source position when the reflection information data exhibits a redundant state, and select one of the plurality of reflection sequence sets as the first set of reflection sequences associated with the first listener position and with the first source position.

[0194] For example, each reflection sequence in each reflection sequence set of multiple reflection sequence sets may, for example, include information about a group of one or more reflection objects of the reflection sequence, where sound waves emitted by an audio source at a source position of the reflection sequence set and perceptible by a listener at a listener position of the reflection sequence set are reflected on the way to the current listener position.

[0195] According to an embodiment, if the reflection information data exhibits a redundant state, the indication may, for example, indicate that the first listener position and the first source position are selected so that the first listener position is adjacent to the second listener position and / or the first source position is adjacent to the second listener position. If the reflection information data exhibits a redundant state, the signal generator 320 may, for example, be configured to determine the first listener position and / or the first source position according to the indication.

[0196] In an embodiment, if the reflection information data exhibits a redundant state, the indication may, for example, indicate that the first listener position and the first source position are selected so that the first listener position is adjacent to the second listener position and the first source position is the same as the second listener position. The signal generator 320 is configured to determine the first listener position and the first source position according to the indication.

[0197] Alternatively, in an embodiment, if the reflection information data exhibits a redundant state, the indication may, for example, indicate that the first listener position and the first source position are selected so that the first listener position is the same as the second listener position and the first source position is adjacent to the second listener position. The signal generator 320 may, for example, be configured to determine the first listener position and the first source position according to the indication.

[0198] According to an embodiment, in a coordinate system, a first position is adjacent to a second position if, in each coordinate direction of the coordinate system, the first position is immediately before the second position, immediately after the second position, or is the same as the second position, and if, in at least one coordinate direction of the coordinate system, the first position and the second position are different from each other, then the first position is adjacent to the second position.

[0199] In an embodiment, the indication may, for example, indicate one of the following:

[0200] -Reflection information data shows a non-redundant state,

[0201] the reflection information data exhibit a first redundancy state such that the first listener position and the first source position should be selected such that the first source position is identical to the second source position and such that the first listener position is adjacent to the second listener position, wherein in a first coordinate direction of the coordinate system the first listener position is immediately before the second listener position, and wherein in a second coordinate direction and in a third coordinate direction of the coordinate system the first listener position is identical to the second listener position,

[0202] the reflection information data exhibit a second redundancy state such that the first listener position and the first source position should be selected such that the first source position is identical to the second source position and such that the first listener position is adjacent to the second listener position, wherein in a second coordinate direction of the coordinate system the first listener position is immediately before the second listener position, and wherein in the first coordinate direction and in a third coordinate direction of the coordinate system the first listener position is identical to the second listener position,

[0203] -The reflection information data exhibits a third redundancy state, so that the first listener position and the first source position should be selected so that the first source position is the same as the second source position, and so that the first listener position is adjacent to the second listener position, wherein in a third coordinate direction of the coordinate system, the first listener position is immediately before the second listener position, and wherein in the first coordinate direction and the second coordinate direction of the coordinate system, the first listener position is the same as the second listener position.

[0204] If the indication indicates the first redundancy state or the second redundancy state or the first redundancy state, the signal generator 320 may, for example, be configured to determine the first listener position and the first source position according to the indication.

[0205] According to an embodiment, each of the first listener position, the first source position, the second listener position and the second source position may, for example, define a position of a voxel among a plurality of voxels within a three-dimensional coordinate system.

[0206] For example, each of the listener position and the source position for each of the plurality of sets of reflection sequences may, for example, define a position of a voxel of a plurality of voxels within a three-dimensional coordinate system.

[0207] In an embodiment, the signal generator 320 may, for example, be configured to generate a binaural signal comprising two binaural channels as the one or more audio output signals.

[0208] Fig.10 A device 400 for encoding one or more audio signals and for generating additional audio information data according to an embodiment is shown.

[0209] The apparatus 400 comprises an audio signal encoder 410 for encoding one or more audio signals to obtain one or more encoded audio signals.

[0210] Furthermore, the apparatus 400 comprises an additional audio information generator 420 for generating additional audio information data, wherein the additional audio information generator 420 exhibits a non-redundant operation mode and a redundant operation mode.

[0211] The additional audio information generator 420 is configured to generate the additional audio information data in a case where the additional audio information generator 420 exhibits a non-redundant operating mode, such that the additional audio information data comprises the second additional audio information.

[0212] In addition, the additional audio information generator 420 is configured to generate additional audio information data when the additional audio information generator 420 exhibits a non-redundant operation mode, so that the additional audio information data does not include the second additional audio information or only includes a portion of the second additional audio information, so that the second additional audio information can be obtained using the additional audio information data together with the first additional audio information.

[0213] According to an embodiment, the additional audio information generator 420 may be, for example, a propagation information generator for generating propagation information data as the additional audio information data. The propagation information generator may, for example, be configured to generate the propagation information data when the propagation information generator exhibits a non-redundant operation mode, so that the propagation information data includes the second additional audio information, which is the second propagation information. In addition, the propagation information generator may, for example, be configured to generate the propagation information data when the propagation information generator exhibits a non-redundant operation mode, so that the propagation information data does not include the second propagation information or includes only a part of the second propagation information, so that the second propagation information can be obtained using the propagation information data together with the first propagation information.

[0214] According to an embodiment, the first propagation information and / or the second propagation information may for example depend on one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment.

[0215] In an embodiment, the propagation information data may include, for example, reflection information data and / or diffraction information data. The first propagation information may include, for example, first reflection information and / or first diffraction information. The second propagation information may include, for example, second reflection information and / or second diffraction information.

[0216] According to an embodiment, the propagation information generator may, for example, be a reflection information generator for generating reflection information data as propagation information data. The reflection information generator may, for example, be configured to generate reflection information data in a case where the reflection information generator exhibits a non-redundant operation mode, so that the reflection information data includes the second reflection information as the second propagation information. In addition, the reflection information generator may, for example, be configured to generate reflection information data in a case where the reflection information generator exhibits a non-redundant operation mode, so that the reflection information data does not include the second reflection information or includes only a portion of the second reflection information, so that the reflection information data can be used together with the first propagation information (which is the first reflection information) to obtain the second reflection information.

[0217] In an embodiment, the first reflection information and / or the second reflection information may, for example, depend on one or more reflections of one or more sound waves propagating along one or more propagation paths in a real listening environment or in a virtual listening environment or in an enhanced listening environment at one or more reflecting objects.

[0218] According to an embodiment, the propagation information generator may, for example, be a diffraction information generator for generating diffraction information data as the propagation information data. The diffraction information generator may, for example, be configured to generate the diffraction information data when the diffraction information generator exhibits a non-redundant operation mode, so that the diffraction information data includes the second diffraction information as the second propagation information. In addition, the diffraction information generator may, for example, be configured to generate the diffraction information data when the diffraction information generator exhibits a non-redundant operation mode, so that the diffraction information data does not include the second diffraction information or includes only a part of the second diffraction information, so that the diffraction information data can be used together with the first propagation information (which is the first diffraction information) to obtain the second diffraction information.

[0219] In an embodiment, the first diffraction information and / or the second diffraction information may, for example, depend on one or more diffraction of one or more sound waves propagating along one or more propagation paths at one or more diffraction objects in a real listening environment or in a virtual listening environment or in an enhanced listening environment.

[0220] According to an embodiment, the propagation information generator may be configured, for example, in a redundant operation mode to generate propagation information data, such that the propagation information data may, for example, indicate one or more propagation sequences to be removed from the first propagation information (which is the first set of propagation sequences), and / or may, for example, indicate one or more propagation sequences to be added to the first set of propagation sequences to obtain the second propagation information (which is the second set of propagation sequences).

[0221] In an embodiment, each propagation sequence of the first set of propagation sequences and the second set of propagation sequences may, for example, indicate a group of one or more reflective objects or a group of one or more diffractive objects.

[0222] In an embodiment, the propagation information generator may be configured, for example, in a non-redundant operation mode to generate the propagation information data, such that the propagation information data may, for example, comprise the second set of propagation sequences.

[0223] According to an embodiment, the first set of propagation sequences may be associated with a first listener position and with a first source position, for example. The second set of propagation sequences may be associated with a second listener position and with a second source position, for example. The first listener position may be different from the second listener position, for example, and / or wherein the first source position may be different from the second source position, for example.

[0224] In an embodiment, the first set of propagation sequences may, for example, be a first set of reflection sequences. The propagation information generator may, for example, be a reflection information generator. The second set of propagation sequences may, for example, be a second set of reflection sequences. The propagation information data may, for example, be reflection information data. Each reflection sequence in the first set of reflection sequences may, for example, contain information about a group of one or more reflection objects of the reflection sequence, wherein a sound wave emitted by an audio source at a first source position and perceivable by a listener at a first listener position is reflected on the way to a current listener position. The reflection information generator may, for example, be configured to generate reflection information data, so that each reflection sequence in the second set of reflection sequences may, for example, contain information about a group of one or more reflection objects of the reflection sequence, wherein a sound wave emitted by an audio source at a second source position and perceivable by a listener at a second listener position is reflected on the way to a current listener position.

[0225] According to an embodiment, the one or more encoded audio signals are associated with audio sources located at source positions of the second set of reflection sequences.

[0226] In an embodiment, the reflection information generator may be configured, for example in a redundant operating mode, to generate an indication of a first listener position and a first source position suitable for determining a first set of reflection sequences.

[0227] According to an embodiment, the reflection information generator may be configured, for example, in a redundant operation mode to generate an indication such that the indication may, for example, indicate selecting a first listener position and a first source position such that the first listener position is adjacent to the second listener position and / or such that the first source position is adjacent to the second listener position.

[0228] In an embodiment, the reflection information generator may be configured, for example, in a redundant operating mode to generate an indication such that the indication may, for example, indicate selecting a first listener position and a first source position such that the first listener position is adjacent to the second listener position and such that the first source position is the same as the second listener position.

[0229] Alternatively, in an embodiment, the reflection information generator may be configured, for example, in a redundant operating mode to generate an indication such that the indication may, for example, indicate selecting a first listener position and a first source position such that the first listener position is the same as the second listener position and such that the first source position is adjacent to the second listener position.

[0230] According to an embodiment, in a coordinate system, a first position is adjacent to a second position if, in each coordinate direction of the coordinate system, the first position is immediately before the second position, immediately after the second position, or is the same as the second position, and if, in at least one coordinate direction of the coordinate system, the first position and the second position are different from each other, then the first position is adjacent to the second position.

[0231] In an embodiment, the reflection information generator may be configured, for example, in a redundant operation mode, to generate an indication, such that the indication may, for example, indicate one of the following:

[0232] -Reflection information data shows a non-redundant state,

[0233] the reflection information data exhibit a first redundancy state such that the first listener position and the first source position should be selected such that the first source position is identical to the second source position and such that the first listener position is adjacent to the second listener position, wherein in a first coordinate direction of the coordinate system the first listener position is immediately before the second listener position, and wherein in a second coordinate direction and in a third coordinate direction of the coordinate system the first listener position is identical to the second listener position,

[0234] the reflection information data exhibit a second redundancy state such that the first listener position and the first source position should be selected such that the first source position is identical to the second source position and such that the first listener position is adjacent to the second listener position, wherein in a second coordinate direction of the coordinate system the first listener position is immediately before the second listener position, and wherein in the first coordinate direction and in a third coordinate direction of the coordinate system the first listener position is identical to the second listener position,

[0235] -The reflection information data exhibits a third redundancy state, so that the first listener position and the first source position should be selected so that the first source position is the same as the second source position, and so that the first listener position is adjacent to the second listener position, wherein in a third coordinate direction of the coordinate system, the first listener position is immediately before the second listener position, and wherein in the first coordinate direction and the second coordinate direction of the coordinate system, the first listener position is the same as the second listener position.

[0236] According to an embodiment, each of the first listener position, the first source position, the second listener position and the second source position may, for example, define a position of a voxel of a plurality of voxels within a three-dimensional coordinate system.

[0237] Fig.11 A system according to another embodiment is presented. The system comprises Fig.10 The system comprises a device 400 for encoding one or more audio signals to obtain one or more encoded audio signals and for generating additional audio information data. Fig. 9 The device 300 is for generating one or more audio output signals from one or more encoded audio signals based on additional audio information data.

[0238] Hereinafter, other specific embodiments are provided.

[0239] More specifically, binary encoding and decoding of metadata is considered.

[0240] The current working draft of the MPEG-I 6DoF Audio specification ("RM0 Draft 1") states that earlySurfaceDataJSON, earlySurfaceConnectedDataJSON and earlyVoxelDataJSON are represented as "a zero-terminated string in ASCII encoding containing a JSON formatted file as a temporary data format". In this input file, we propose to replace this temporary data format with a binary data format using an encoding method that significantly reduces the bitstream size.

[0241] This core experiment is based on the first draft version of RM0. It aims to replace the JSON formatted early reflection metadata with a binary encoding format. By applying specific techniques, a significant reduction in the size of the early reflection payload is achieved while introducing insignificant quantization errors.

[0242] Techniques applied to reduce payload size include:

[0243] 1. Data consolidation: Removed variables (earlySurfaceConnectedData) no longer used by RefSoft renderer.

[0244] 2. Coordinate system: The unit normal vector to the reflection plane is transferred in spherical coordinates rather than Cartesian coordinates to reduce the number of coefficients from 3 to 2.

[0245] 3. Quantization: The coefficients defining the reflection plane are quantized at high resolution (quasi-lossless coding).

[0246] 4. Entropy coding: A general coding scheme based on a codebook is used for entropy coding of the transmitted symbols. The applied method is particularly beneficial for data sequences with a very large number of symbols (while also suitable for a small number of symbols).

[0247] 5. Inter-voxel redundancy reduction: The similarity of voxel data of voxel neighbors is exploited to further reduce the bitstream size. A differential method is used, in which the difference between the current voxel data set and the neighboring voxel data set is encoded.

[0248] The decoder is simplified since the parsing step of the JSON data is no longer required, while the runtime complexity of the renderer is not affected by the proposed changes.

[0249] Furthermore, the proposed replacement also reduces the renderer's dependencies as well as the encoder's dependencies, since there is no need to generate and parse JSON files anymore.

[0250] For all "Test 1" and "Test 2" scenarios, the proposed encoding method reduces the overall bitstream size by 21.33% on average compared to P13. Considering the scenario with only reflected mesh data, the proposed encoding method reduces the overall bitstream size by 28.91% on average compared to P13.

[0251] In the following, information about addition / replacement is considered.

[0252] The encoding method presented in this core experiment is intended to replace the main part of payloadEarlyReflections(). It is intended to replace the corresponding payload handler for packets of type PLD_EARLY_REFLECTIONS in the reference software accordingly.

[0253] In the following, additional technical information is provided.

[0254] Specifically, it is proposed to remove unused variables

[0255] The RM0 bitstream parser generates data structures earlySurfaceData and earlySurfaceConnectedData according to the bitstream variables earlySurfaceDataJSON and earlySurfaceConnectedDataJSON. The data defines the reflection plane of the static scene geometry and the triangles belonging to the connected surface area. The motivation for splitting all triangle sets belonging to the reflection plane into several groups of connected areas is to allow the renderer to check only the sub-sets during the visibility test. However, the reference software implementation no longer utilizes this unique information. Internally, the Intel Embree library uses its own acceleration method (bounding volume hierarchy data structure) for fast ray tracing.

[0256] Therefore, it is proposed to simplify these data structures by combining them into a single data structure without the connected surface information:

[0257] Table - earlySurfaceData() data structure

[0258]

[0259] In the following, quantization is considered.

[0260] Rather than transmitting the Cartesian coordinates of the unit normal vector N0, it is more efficient to transmit spherical coordinates, since one of the values ​​(the distance) is constant and does not need to be transmitted:

[0261]

[0262] r dist =1

[0263] Proposed to use 12 bits to quantize azimuth And the elevation angle θ is quantized using 11 bits ele , as follows:

[0264]

[0265] and the elevation angle of the surface normal N0, as follows:

[0266]

[0267] This quantization scheme ensures that integer multiples of 5° and various delimiters of 360° that are powers of 2 are directly on the quantization grid. Due to the high resolution, the resulting 4032 quantization steps in azimuth and 2017 in elevation can be considered quasi-lossless.

[0268] For quantification of surface distances d, we propose a 1 mm resolution, which is also the same resolution used to transmit scene geometry data.

[0269] The actual number of bits used to transmit these values ​​depends on the entropy coding scheme described in the following section.

[0270] In the following, entropy coding according to a specific embodiment is considered.

[0271] If the symbols are not evenly distributed, entropy coding can be used to reduce the amount of bits needed to transmit the data. A widely used method for entropy coding is Huffman coding, which uses smaller codewords for more frequent symbols and longer codewords for less frequent symbols, thereby generating a smaller average word size. Recently, arithmetic coding has become popular, which encodes the entire information text at once. For example, for the encoding of directional data, an adaptive arithmetic coding mechanism is used. This adaptive method is particularly advantageous if the symbol distribution changes steadily over time.

[0272] As far as the early reflection metadata is concerned, we cannot make any assumptions about the temporal behavior of the symbol distribution (e.g. some symbols appear more frequently at the beginning of a transmission and others appear more frequently at the end of a transmission). It is more reasonable to assume that the symbol distribution is fixed and can be determined during the initialization of the encoder. Moreover, adjusting the symbol distribution at run time and using a symbol distribution that deviates from the a priori known symbol distribution would actually undermine the theoretical advantages of the adaptive arithmetic coding approach.

[0273] For this reason, it is proposed to use classical Huffman codes for entropy coding of early reflection metadata. This requires the use of a predefined codebook, the codebook used, or a binary decoding tree to be transmitted together with the corresponding symbol list. The corresponding symbol list can be efficiently generated by a recursive algorithm: the recursive algorithm traverses the decoding tree and encodes the leaves (i.e., valid codewords) with "1" and encodes the branches with "0". If the current word is not a valid codeword, that is, the algorithm is at a branch of the decoding tree, then 2 recursions are performed: one for the left side of the current word extended by "0" and one for the right side of the current word extended by "1". The following pseudo code shows the encoding algorithm for the decoding tree:

[0274] Using a predefined codebook is actually one of three options, namely, using a predefined codebook, or using a codebook containing a codeword list and a symbol list, or using a decoding tree and a symbol list.

[0275]

[0276] The algorithm also generates a list of all symbols in tree traversal order. The same mechanism can be used on the decoder side to extract the decoding tree topology and valid codewords:

[0277]

[0278]

[0279] This makes the encoding of the decoding tree extremely efficient since only a single bit is consumed per codeword and per branch.

[0280] In addition to the topology of the decoding tree, the symbol list also needs to be transmitted in tree traversal order for the complete transmission of the codebook.

[0281] In some cases, transmitting the codebook in addition to the symbols may make the bitstream even larger than a simple fixed-length encoding. Therefore, we introduce a new general method for transmitting data using codebooks. The method we propose utilizes variable-length encoding or fixed-length encoding using the encoding scheme described above. In the latter case, only the word size, i.e. the number of bits per codeword, has to be transmitted, rather than the complete codebook. Alternatively, a common offset of the integer value of the symbol can be given in the bitstream if the difference with the offset results in a smaller word length. The following function parses such a general codebook and returns the data structure of the current codebook instance:

[0282]

[0283] In this embodiment, the keyword "Bitarray" is used as an alias for a bit sequence of a specific length. In addition, the keyword "append()" indicates a method that extends the length of an array by adding one or more elements at the end.

[0284] The tree traversal function that is performed recursively is defined as follows:

[0285]

[0286] Since they have different sign distributions, we propose to use separate codebooks for the following arrays:

[0287] earlySurfaceLengthFaceIdx

[0288] earlySurfaceFaceIdx

[0289] ·earlySurfaceAzi

[0290] ·earlySurfaceEle

[0291] earlySurfaceDist

[0292] earlyVoxelL (see next section)

[0293] earlyVoxelS (see next section)

[0294] earlyVoxelIndicesRemovedDiff (see next section)

[0295] earlyVoxelNumPaths (see next section)

[0296] earlyVoxelOrder (see next section)

[0297] earlyVoxelSurf (see next section)

[0298] In the following, inter-voxel redundancy reduction according to certain embodiments is described.

[0299] The early reflection voxel database earlyVoxelDatabase[l][s] stores a list of reflection sequences that may be visible to a source in a voxel with index s and a listener in a voxel with index l. In many cases, the lists of reflection sequences will be very similar for adjacent voxels. By reducing this inter-voxel redundancy, the bitstream size can be significantly reduced.

[0300] The proposed inter-voxel redundancy reduction uses 4 operation modes, which are signaled by the bitstream variable earlyVoxelMode[v]. In mode 0 ("no reference"), the list of reflection sequences for source voxels earlyVoxelS[v] and listener voxels earlyVoxelL[v] is transmitted as an array with path index p and order index o using a general codebook of variables earlyVoxelNumPaths[v], earlyVoxelOrder[v][p] and earlyVoxelSurf[v][p][o]. In other operation modes, the difference between the list of reference and current reflection sequences is transmitted.

[0301] In mode 1 ("x-axis reference"), a list of reflection sequences of the current source voxel and listener voxel neighbors in the negative x-axis direction is used as reference. A list of indices (which specify the entries of the reference list that need to be removed) and a list of additional reflection sequences are transmitted.

[0302] Mode 2 ("y-axis reference") differs from Mode 1 in that the listener voxel neighbors in the negative y-axis direction are used.

[0303] Mode 3 ("z-axis reference") differs from Mode 1 in that the listener voxel neighbors in the negative z-axis direction are used.

[0304] If a zero-terminated list of differences earlyVoxelIndicesRemovedDiff[v] is transmitted, then a list of indices earlyVoxelIndicesRemoved[v] that specifies the entries of the reference list that need to be removed can be encoded more efficiently. This reduces entropy because smaller values ​​become more likely and larger values ​​become less likely, resulting in a more distinct distribution. The conversion is performed via accumulation:

[0305]

[0306] Hereinafter, the syntax of the general codebook is described.

[0307] Some payloads such as payloadEarlyReflections() utilize separate codebooks, which are defined within the bitstream using the following syntax:

[0308] Table—Grammar of genericCodebook()

[0309]

[0310] The codeword list "codeList" is transmitted using the following recursive tree traversal algorithm, where the keyword "Bitarray" is used as an alias for a bit sequence of a specific length. In addition, the keyword "append()" indicates a method that extends the length of the array by adding one or more elements at the end:

[0311] Table — Syntax of traverseTreeDecode()

[0312]

[0313] An instance of such a codebook "exampleCodebook" is created as follows:

[0314] exampleCodebook=genericCodebook();

[0315] In addition to the data fields of the returned data structure, the general codebook has a method "get_symbol()" that reads a valid codeword from the bitstream, i.e., the nth element of codeList[], and returns the corresponding symbol, i.e., symbolList[n]. The usage instructions of this method are as follows:

[0316] exampleVariable=exampleCodebook.get_symbol();

[0317] In the following, the proposed syntax for early reflection payload is introduced.

[0318] Table — Syntax of payloadEarlyReflections()

[0319]

[0320]

[0321] Table—Syntax of earlySurfaceData()

[0322]

[0323] Table—Syntax of earlyVoxelData()

[0324]

[0325]

[0326] In the following, the proposed data structure, namely the early reflection payload data structure, is introduced.

[0327] earlyTriangleCullingDistanceOrder1 The triangle culling distance for first-order reflections.

[0328] earlyTriangleCullingDistanceOrder2 The triangle culling distance of the second-order reflection.

[0329] earlySourceCullingDistanceOrder1 Source culling distance for first order reflections.

[0330] earlyTriangleCullingDistanceOrder2 Source culling distance for second-order reflections.

[0331] earlyVoxelGridOriginX The x-component of the Cartesian coordinate of the voxel grid origin [0,0,0].

[0332] earlyVoxelGridOriginY The y component of the Cartesian coordinate of the voxel grid origin [0,0,0].

[0333] earlyVoxelGridOriginZ The z component of the Cartesian coordinate of the voxel grid origin [0,0,0].

[0334] earlyVoxelGridPitchX The voxel grid pitch (voxel width) along the x-axis.

[0335] earlyVoxelGridPitchY The voxel grid pitch (voxel length) along the y-axis.

[0336] earlyVoxelGridPitchZ The voxel grid pitch (voxel height) along the z-axis.

[0337] earlyVoxelGridShapeX The number of voxels along the x-axis.

[0338] earlyVoxelGridShapeY The number of voxels along the y-axis.

[0339] earlyVoxelGridShapeZ The number of voxels along the z-axis.

[0340] earlyHasSurfaceData Flag indicating the presence of earlySurfaceData.

[0341] earlySurfaceDataLength The length of earlySurfaceData in bits.

[0342] earlyHasVoxelData Flag indicating the presence of earlyVoxelData.

[0343] earlyVoxelDataLength The length of earlySurfaceData in bits.

[0344] earlySurfaceDistOffset The offset of earlySurfaceDist in mm.

[0345] numberOfSurfaces The number of surfaces.

[0346] earlySurfaceLengthFaceIdx The array length of earlySurfaceFaceIdx.

[0347] earlySurfaceFaceIdx List of triangle IDs.

[0348] earlySurfaceAzi An array with azimuths specifying the surface normal in spherical coordinates (Hesse normal form).

[0349] earlySurfaceEle An array with elevation angles specifying the surface normal in spherical coordinates (Hesse normal form).

[0350] earlySurfaceDist An array with distance values ​​(in Hesse normal form).

[0351] numberOfVoxelPairs The number of source & listener voxels paired with available voxel data

[0352] earlyVoxelL An array with the listener voxel indices.

[0353] earlyVoxelS An array with the source voxel indices.

[0354] earlyVoxelMode An array specifying the encoding mode of the voxel data.

[0355] earlyVoxelIndicesRemovedDiff A differently encoded removal list that specifies the indices of the reference reflection sequence list that should be removed.

[0356] earlyVoxelNumPaths The number of reflection paths.

[0357] earlyVoxelOrder 2D array specifying the reflection order.

[0358] earlyVoxelSurf The sequence of reflections given as a 3D array of surface indices.

[0359] In the following, a renderer stage that considers early reflections is proposed and terminology and definitions are provided.

[0360] Voxel grid:

[0361] The renderer uses voxel data to speed up computationally complex visibility checks of reflected sound propagation paths. The scene is rasterized into a regular grid with a grid spacing that can be defined independently for each dimension. Each voxel is identified by a unique voxel ID, and a sparse database is used to store precomputed data for a given source / listener voxel pair. The relevant variables and data structures are:

[0362] ·earlyVoxelGridOriginX

[0363] earlyVoxelGridOriginY

[0364] earlyVoxelGridOriginZ

[0365] earlyVoxelGridPitchX

[0366] earlyVoxelGridPitchY

[0367] earlyVoxelGridPitchZ

[0368] earlyVoxelGridShapeX

[0369] earlyVoxelGridShapeY

[0370] earlyVoxelGridShapeZ

[0371] These variables are voxel coordinates V = [vx, vy, vz] with 3 integer components. T For any point P = [px, py, pz] in the scene T , the corresponding voxel coordinates are calculated to the nearest integer by the following rounding operation:

[0372] v x =round((p x -earlyVoxelGridOriginX) / earlyVoxelGridPitchX) (1)

[0373] v y =round((p y -earlyVoxelGridOriginY) / earlyVoxelaagridPitchY) (2)

[0374] v z =round((p z -earlyVoxelGridOriginZ) / earlyVoxelaagridPitchZ) (3) Voxel coordinates can be converted to voxel indices:

[0375] n=v x +earlyVoxelGridShapeX*(v y +earlyVoxelGridShapeY*v z ) (4)

[0376] This representation is used, for example, in the sparse voxel database earlyVoxelDatabase[l][s][p] for the listener voxel ID l and the source voxel ID s.

[0377] Cull distance:

[0378] The encoder can use source and / or triangle distance culling to speed up the pre-computation of voxel data. The culling distance is encoded in the bitstream to allow the renderer to smoothly fade reflections that reach the used culling threshold. The relevant variables and data structures are:

[0379] ·earlyTriangleCullingDistanceOrder1

[0380] ·earlyTriangleCullingDistanceOrder2

[0381] ·earlySourceCullingDistanceOrder1

[0382] ·earlySourceCullingDistanceOrder2

[0383] Surface data:

[0384] Surface data is the geometric data that defines the reflection plane, and the sound is reflected on the reflection plane. The relevant variables and data structures are:

[0385] earlySurfaceIdx[s];

[0386] ·earlySurfaceFaceIdx[s][f];

[0387] earlySurface_N0[s]

[0388] ·earlySurface_d[s]

[0389] The surface index earlySurfaceIdx[s] identifies the surface and is referenced by the sparse voxel database earlyVoxelDatabase[l][s][p]. The triangle ID list earlySurfaceFaceIdx[s][f] defines the triangles of the static mesh belonging to the surface. One of these triangles must be hit for the visibility test of the specular reflection to succeed. The reflection plane for each surface is given in Hesse normal form using the surface normal N0 and the surface distance d transformed as follows:

[0390] int max_steps_azi=1<<12;

[0391] int max_steps_ele = 1<<11;

[0392] int num_steps_azi=144*(max_steps_azi / 144);

[0393] int num_steps_ele=72*(max_steps_ele / 72);

[0394] int shift_ele=num_steps_ele / 2;

[0395] float quant2azi=double(2.0*M_PI) / double(num_steps_azi);

[0396] float quant2ele=double(M_PI) / double(num_steps_ele);

[0397] float quant2dist = 0.001f;

[0398] for(int s=0; s <numberOfSurfaces;s++){

[0399] earlySurfaceIdx[s] = s;

[0400] float azi=earlySurfaceAzi[s]*quant2azi;

[0401] float ele=(earlySurfaceEle[s]-shift_ele)*quant2ele;

[0402] earlySurface_N0[s][0]=-1.0*sin(azi)*cos(ele);

[0403] earlySurface_N0[s][1]=sin(ele);

[0404] earlySurface_N0[s][2]=-1.0*cos(azi)*cos(ele);

[0405] earlySurface_d[s]=(earlySurfaceDist[s]+dist_offset)*quant2dist;

[0406] }

[0407] Voxel data

[0408] The early reflection voxel data is a sparse voxel database that contains a list of reflection sequences of potentially visible image sources for a given source and listener voxel pair. The entries of the database may be undefined for a given source and listener voxel pair not specified in the bitstream, it may be an empty list, or it may contain a list of surface connection IDs. The relevant variables and data structures are:

[0409] numberOfVoxelPairs

[0410] ·earlyVoxelL[v]

[0411] ·earlyVoxelS[v]

[0412] earlyVoxelMode[v]

[0413] ·earlyVoxelIndicesRemovedDiff[v][k]

[0414] earlyVoxelNumPaths[v]

[0415] ·earlyVoxelOrder[v][p]

[0416] ·earlyVoxelSurf[v][p][o]

[0417] The sparse voxel database earlyVoxelDatabase[l][s][p] is derived from these variables using the following algorithm:

[0418] int delta_x=voxelCoordinateToVoxelIndex({1,0,0});

[0419] int delta_y=voxelCoordinateToVoxelIndex({0,1,0});

[0420] int delta_z=voxelCoordinateToVoxelIndex({0,0,1});

[0421] int delta_list[4]={0,-delta_x,-delta_y,-delta_z};

[0422]

[0423]

[0424] In this algorithm, the function voxelCoordinateToVoxelIndex() represents the conversion of voxel coordinates to voxel indices. The keyword PathList identifies a list of integer arrays, which can be modified by the method append() to add elements at the end of the list and the method erase() to remove list elements at a given position. In addition, the function shortlex_sort() represents a sorting function that sorts a given list of reflection sequences in shortlex order.

[0425] Complexity Assessment

[0426] The decoder is simplified because the parsing step of the JSON data is no longer required, while the runtime complexity of the renderer is not affected by the proposed changes.

[0427] Proof of Advantages

[0428] To verify that the proposed method works correctly and to demonstrate its technical advantages, we encoded all “Test 1” and “Test 2” scenes and compared the size of the early reflection metadata with the encoding results of the P13 encoder.

[0429] Data Compression

[0430] Table 2 lists the size of payloadEarlyReflection for the P13 encoder ("old size / byte") and the variant of the P13 encoder ("new size / byte") using the proposed encoding method. The last row lists the compression ratio obtained, i.e. the ratio of the old payload size to the new payload size.

[0431] In all cases, the proposed method generates smaller payload sizes. For scene objects with reflections All scenes, i.e. scenes with mesh data, achieve a compression ratio greater than 10. For some scenes ("Singer in the Lab" and "Virtual Basketball"), compression ratios close to or even greater than 100 were achieved.

[0432] Table - PayloadEarlyReflection size comparison

[0433]

[0434]

[0435] In the following, the overall bitstream saving is considered.

[0436] The following table lists the total bitstream size savings (in percent). On average, the total bitstream size is reduced by 21.33%. Considering only the scenarios with mesh data, the total bitstream size is reduced by 28.91% on average.

[0437] Table - Total bitstream size savings

[0438]

[0439] Data verification and quantification error

[0440] The following table lists the results of our data validation tests on an extended test set, which additionally includes all "test 4" scenes plus other scenes that did not make it into the official test repository, where we compare the decoded metadata such as earlySurfaceData and earlyVoxelData with the output of the P13 decoder. For the P13 payload, the concatenated surface data is combined with the surface data to be able to compare it with the new encoding method. The validation result "same structure" means that both payloads have the same reflective surface and the data differs only due to the expected quantization error.

[0441] For all scenes, the decoded earlyVoxelData is the same, and the decoded earlySurfaceData is the same or structurally identical.

[0442] Table - Validation of transmitted data

[0443]

[0444] The following table lists the minimum, mean, median and maximum quantization errors in mm of the transmitted plane normal N0 after conversion to Cartesian coordinates. The maximum quantization error of 1.095 mm corresponds to an angular deviation of 0.063°. With a resolution of 0.088° per quantization step and therefore a maximum quantization error of 0.044° per axis, the observed results are highly consistent with the theoretical values.

[0445] For the surface normal vector N0, the maximum angular deviation of 0.063° is so small that the transmission can be considered quasi-lossless.

[0446] Table - Quantization error of the surface normal unit vector (in mm)

[0447]

[0448]

[0449] The following table lists the minimum, mean, median and maximum quantization errors of the transmitted planar distance in mm. The observed maximum deviation of 0.519 mm is highly consistent with the expected maximum of 0.5 mm at a resolution of 1 mm per quantization step. The overshoot can be explained by the limited accuracy of the single precision floating point variables used, which cannot provide sufficient sub-millimeter resolution for large scenes such as "park", "parking lot" and "entertainment".

[0450] For the surface distance d, the maximum deviation of 0.519 mm is so small that the transmission can be considered quasi-lossless.

[0451] Table - Quantization error of surface distance (in mm)

[0452]

[0453] In an embodiment, a binary encoding method for earlySurfaceDat() and earlyVoxelData() as part of the early reflections metadata in payloadEarlyReflections() is provided. For a test set of 30 AR and VR scenes, we compared the decoded data with that decoded by the P13 decoder and observed only the expected quantization errors. The quantization errors of the surface data are so small that the transmission can be considered quasi-lossless. The transmitted voxel data is identical.

[0454] In all cases, the proposed method generates a smaller payload size. For all scenes with reflective scene objects, i.e., scenes with mesh data, a compression ratio greater than 10 is achieved. For some scenes ("Singer in the Lab" and "Virtual Basketball"), a compression ratio close to or even greater than 100 is achieved. For all "Test 1" and "Test 2" scenes, the proposed encoding method reduces the overall bitstream size by an average of 21.33% compared to P13. Considering scenes with only reflective mesh data, the proposed encoding method reduces the overall bitstream size by an average of 28.91% compared to P13.

[0455] The proposed encoding method does not affect the runtime complexity of the renderer.

[0456] Furthermore, the proposed replacement also reduces the library dependencies of the reference software, since there is no need to generate and parse JSON files anymore.

[0457] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent descriptions of corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent descriptions of features of corresponding blocks or items or corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices, such as, for example, microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed by such devices.

[0458] Depending on certain implementation requirements, embodiments of the present invention may be implemented by hardware or software, or at least partially in hardware or at least partially in software. Embodiments may be implemented using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or are capable of cooperating) with a programmable computer system to cause the corresponding method to be performed. Thus, the digital storage medium may be computer readable.

[0459] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0460] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0461] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0462] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0463] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[0464] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example via the Internet.

[0465] A further embodiment comprises processing means, for example a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0466] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0467] Another embodiment according to the invention comprises an apparatus or a system configured to transfer (for example electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device or the like. The apparatus or the system may, for example, comprise a file server for transferring the computer program to the receiver.

[0468] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, these methods are preferably performed by any hardware device.

[0469] The devices described herein may be implemented using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0470] The methods described herein may be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0471] The embodiments described above are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the configurations and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended only to be limited by the scope of the following patent claims, rather than by the specific details rendered as the description and explanation of the embodiments herein.

Claims

1. A device (100) for generating one or more audio output signals from one or more encoded audio signals, wherein the device (100) comprises: at least one entropy decoding module (110; 116, 118) for decoding the encoded additional audio information to obtain decoded additional audio information when the encoded additional audio information is entropy encoded, and A signal processor (120) is used to generate one or more audio output signals based on the one or more encoded audio signals and based on the decoded additional audio information.

2. The device (100) as claimed in claim 1, The device (100) further comprises: at least one non-entropy decoding module (111) for decoding the encoded additional audio information to obtain decoded additional audio information when the encoded additional audio information is not entropy encoded, and A selector (115) is used to select one of at least one entropy decoding module (110; 116, 118) and at least one non-entropy decoding module (111) for decoding the encoded additional audio information according to whether the encoded additional audio information is entropy encoded.

3. The device (100) as claimed in claim 1 or 2, The additional audio information encoded therein includes augmented reality data or virtual reality data.

4. The device (100) as claimed in one of the preceding claims, The additional audio information encoded therein depends on the real listening environment or on a virtual listening environment or on an enhanced listening environment.

5. The device (100) as claimed in claim 4, The encoded additional audio information includes propagation information according to one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment, a virtual listening environment, or an enhanced listening environment.

6. The device (100) as claimed in claim 5, The propagation information is reflection information based on one or more reflections of one or more sound waves propagating along one or more propagation paths at one or more reflection objects in a real listening environment, a virtual listening environment, or an enhanced listening environment.

7. The device (100) as claimed in claim 5, The propagation information is diffraction information based on one or more diffraction of one or more sound waves propagating along one or more propagation paths at one or more diffraction objects in a real listening environment, a virtual listening environment, or an enhanced listening environment.

8. The device (100) as claimed in one of the preceding claims, The extra audio information encoded therein contains data for rendering early reflections. The signal processor (120) is configured to generate one or more audio output signals based on the data for rendering the early reflections.

9. The device (100) as claimed in one of the preceding claims, The signal processor (120) is configured to generate a binaural signal including two binaural channels as one or more audio output signals.

10. The device (100) as claimed in one of the preceding claims, At least one of the entropy decoding modules (110; 116, 118) comprises a Huffman decoding module (116) for decoding the encoded additional audio information when the encoded additional audio information is Huffman encoded.

11. The device (100) as claimed in one of the preceding claims, At least one of the entropy decoding modules (110; 116, 118) comprises an arithmetic decoding module (118) for decoding the encoded additional audio information when the encoded additional audio information is arithmetically encoded.

12. The device (100) as claimed in claim 2 and claim 10 and claim 11, The selector (115) is configured to select at least one non-entropy decoding module (111) and one of the Huffman decoding module (116) and the arithmetic decoding module (118) for decoding the encoded additional audio information.

13. The device (100) according to one of the preceding claims, further according to claim 2, At least one of the non-entropy decoding modules (111) comprises a fixed length decoding module, which is used to decode the encoded additional audio information when the encoded additional audio information is fixed length encoded.

14. The device (100) according to one of the preceding claims, further according to claim 2, wherein the device (100) is configured to receive selection information, and The selector (115) is configured to select one of at least one entropy decoding module (110; 116, 118) and at least one non-entropy decoding module (111) according to the selection information.

15. The device (100) as claimed in one of the preceding claims, The device (100) is configured to receive a codebook or a coding tree on which the encoded additional audio information depends, and At least the entropy decoding module (110; 116, 118) is configured to decode the encoded additional audio information using a codebook or using a coding tree.

16. The device (100) as claimed in claim 15, wherein the device (100) is configured to receive an encoding of a structure of a coding tree on which the encoded additional audio information depends, wherein at least the entropy decoding module (110; 116, 118) is configured to reconstruct a plurality of codewords of the coding tree according to the structure of the coding tree, and At least the entropy decoding module (110; 116, 118) is configured to decode the encoded additional audio information using the codewords of the coding tree.

17. The device (100) according to any one of claims 1 to 14, The device (100) further comprises a memory storing a codebook or a coding tree, At least the entropy decoding module (110; 116, 118) is configured to decode the encoded additional audio information using a codebook or using a coding tree.

18. The device (100) as claimed in one of the preceding claims, wherein the device (100) is configured to receive encoded additional audio information comprising a plurality of transmitted symbols and an offset value, and At least one non-entropy decoding module (111) is configured to decode the encoded additional audio information using the plurality of transmitted symbols and using the offset value.

19. The device (100) according to any of the preceding claims, further according to claim 8, wherein the data used to render the early reflections comprises information about the position of one or more walls, the one or more walls being one or more real walls or virtual walls in the environment, The signal processor (120) is configured to generate one or more audio output signals based on information about the position of one or more walls.

20. The device (100) as claimed in claim 19, wherein the information about each of the one or more walls comprises information about an azimuth and / or an elevation of the wall, wherein the azimuth of the wall is entropy coded and / or the elevation of the wall is entropy coded, and One or more entropy decoding modules of at least one entropy decoding module (110; 116, 118) are configured to decode the entropy encoded azimuth angle of the wall and / or the entropy encoded elevation angle of the wall.

21. The device (100) according to claim 20, further according to any one of claims 15 to 17, That The one or more of the at least one entropy decoding modules (110; 116, 118) are configured to decode the entropy encoded azimuth angle of the wall and / or the entropy encoded elevation angle of the wall using a codebook or a coding tree.

22. The device (100) as claimed in one of the preceding claims, wherein the encoded additional audio information comprises voxel position information, wherein the position information comprises information regarding one or more positions of one or more voxels of the plurality of voxels within the three-dimensional coordinate system, The signal processor (120) is configured to generate one or more audio output signals based on the voxel position information.

23. The device (100) according to one of the preceding claims, At least one of the entropy decoding modules (110; 116, 118) is configured to decode the entropy-encoded encoded additional audio information, wherein the entropy-encoded encoded additional audio information comprises at least one of: - a list of triangle indices, - the array length of the list of triangle indices, - an array with azimuth angles specifying the surface normal in spherical coordinates, - an array with elevation angles, specifying the surface normal in spherical coordinates, - an array with distance values, - an array with the positions of the listeners, - an array with the positions of one or more sound sources, - a remove list or remove set, specifying the indices of the reflection sequences in the reflection sequence set that should be removed or a list of reference reflection sequences that should be removed, - the number of reflection sequences or the number of reflection paths, - array, specifying the reflection order, -Reflection sequence.

24. An apparatus (200) for encoding one or more audio signals and additional audio information, wherein the apparatus (200) comprises: an audio signal encoder (210) for encoding one or more audio signals to obtain one or more encoded audio signals, and At least one entropy coding module (220) is configured to encode the additional audio information using entropy coding to obtain encoded additional audio information.

25. The device (200) as claimed in claim 24, The device (200) further comprises: at least one non-entropy coding module (221) for encoding the additional audio information to obtain encoded additional audio information, and A selector (215) is used to select one of at least one entropy coding module (220) and at least one non-entropy coding module (221) for encoding the additional audio information according to the distribution of symbols in the additional audio information to be encoded.

26. The device (200) according to claim 24 or 25, The additional audio information encoded therein includes augmented reality data or virtual reality data.

27. The device (200) according to any one of claims 24 to 26, The additional audio information encoded therein depends on the real listening environment or on a virtual listening environment or on an enhanced listening environment.

28. The device (200) as claimed in claim 27, The encoded additional audio information includes propagation information according to one or more propagations of one or more sound waves along one or more propagation paths in a real listening environment, a virtual listening environment, or an enhanced listening environment.

29. The device (200) as claimed in claim 28, The propagation information is reflection information based on one or more reflections of one or more sound waves propagating along one or more propagation paths at one or more reflection objects in a real listening environment, a virtual listening environment, or an enhanced listening environment.

30. The device (200) as claimed in claim 28, The propagation information is diffraction information based on one or more diffraction of one or more sound waves propagating along one or more propagation paths at one or more diffraction objects in a real listening environment, a virtual listening environment, or an enhanced listening environment.

31. The device (200) according to any one of claims 24 to 30, The extra audio information encoded therein contains data for rendering early reflections.

32. The device (200) according to any one of claims 24 to 31, At least one of the entropy coding modules (220) includes a Huffman coding module (226) configured to encode the additional audio information using Huffman coding.

33. The device (200) according to any one of claims 24 to 32, At least one of the entropy coding modules (220) includes an arithmetic coding module (228) configured to encode the additional audio information using arithmetic coding.

34. The device (200) as claimed in claim 25 and claim 32 and claim 33, The selector (215) is configured to select at least one non-entropy coding module (221) and one of a Huffman coding module (226) and an arithmetic coding module (228) for encoding the additional audio information.

35. The device (200) of any one of claims 24 to 34, further according to claim 25, At least one of the non-entropy coding modules (221) includes a fixed length coding module for encoding additional audio information.

36. The device (200) of any one of claims 24 to 35, further according to claim 25, The device (200) is configured to generate selection information indicating one of at least one entropy coding module (220) and at least one non-entropy coding module (221) that has been used to encode the additional audio information.

37. The device (200) according to any one of claims 24 to 36, The device (200) is configured to transmit a codebook or a coding tree that has been used to encode the additional audio information.

38. The device (200) as claimed in claim 37, The device (200) is configured to transmit an encoding of the structure of a coding tree on which the encoded additional audio information depends.

39. The device (200) according to any one of claims 24 to 36, The device (200) further comprises a memory having a codebook or a coding tree stored therein, At least the entropy coding module (220) is configured to encode the additional audio information using a codebook or using a coding tree.

40. The device (200) according to any one of claims 24 to 39, At least one entropy coding module (220) is configured to encode the additional audio information such that the encoded additional audio information includes a plurality of transmitted symbols and an offset value.

41. The device (200) of any one of claims 24 to 40, further according to claim 31, The data used to render the early reflections includes information about the position of one or more walls, which are one or more real walls or virtual walls in the environment.

42. The device (200) as claimed in claim 41, wherein the information about each of the one or more walls comprises information about an azimuth and / or an elevation of the wall, wherein the azimuth of the wall is entropy coded and / or the elevation of the wall is entropy coded, and One or more entropy encoding modules of the at least one entropy encoding module (220) are configured to encode the additional audio information, so that the encoded additional audio information includes the entropy encoded azimuth angle of the wall and / or the entropy encoded elevation angle of the wall.

43. The device (200) of claim 42, further according to any one of claims 37 to 39, The one or more entropy coding modules are configured to use a codebook or a coding tree to encode the entropy-coded azimuth angle of the wall and / or the entropy-coded elevation angle of the wall.

44. The device (200) according to any one of claims 24 to 43, The encoded additional audio information includes voxel position information, wherein the position information includes information about one or more positions of one or more voxels of the plurality of voxels within the three-dimensional coordinate system.

45. The device (200) according to any one of claims 24 to 44, At least one of the entropy coding modules (220) is configured to encode additional audio information using entropy coding, wherein the encoded additional audio information includes at least one of the following: a list of triangle indices, The array length of the list of triangle indices, An array with azimuth angles specifying the surface normal in spherical coordinates, An array with elevation angles specifying the surface normal in spherical coordinates, an array with distance values, an array with the positions of the listeners, an array with the positions of one or more sound sources, A remove list or remove set specifies the index of the reflection sequence in the reflection sequence set that should be removed or a list of reference reflection sequences that should be removed. The number of reflection sequences or the number of reflection paths, An array specifying the reflection order, Reflection sequence.

46. ​​A system comprising: The device (200) as claimed in any one of claims 24 to 45, for encoding one or more audio signals and additional audio information to obtain one or more encoded audio signals and encoded additional audio information, and 24. The device (100) as claimed in claim 1, for generating one or more audio output signals from one or more encoded audio signals and depending on the encoded additional audio information.

47. A method for generating one or more audio output signals from one or more encoded audio signals, wherein the method comprises: When the encoded additional audio information is entropy encoded, decoding the encoded additional audio information to obtain decoded additional audio information, and: One or more audio output signals are generated from the one or more encoded audio signals and from the decoded additional audio information.

48. A method for encoding one or more audio signals and additional audio information, wherein the method comprises: encoding the one or more audio signals into one or more encoded audio signals, and The additional audio information is encoded using entropy coding to obtain encoded additional audio information.

49. A computer program for carrying out the method of claim 47 or 48 when executed on a computer or a signal processor.

Images