Audio rendering suitable for reverberation chamber

Abstract

And an audio processor for performing audio rendering by generating rendering parameters, the rendering parameters determining that speaker signals to be reproduced by the set of speakers are derived from the audio signal. The audio processor is configured to obtain the reverberation effect information and perform a gain adjustment in order to determine a gain for generating a speaker signal for the speaker from the audio signal in accordance with the listener position. The audio processor is configured to, in a gain adjustment, use a roll-off gain compensation function for the at least one speaker in accordance with the reverberation effect information for mapping a listener-to-speaker distance of the at least one speaker onto a listener-to-speaker distance compensation gain for the at least one speaker, the roll-off gain compensation function for the listener-to-speaker distance compensation gain for the at least one speaker. The compensated roll-off is monotonically shallower as the listener-to-speaker distance increases.

Description

[0001] manual Technical Field

[0002] Embodiments according to the invention relate to an audio processor, a system, a method and a computer program for audio rendering, such as user-adaptive loudspeaker rendering for a reverberant room. Background Art

[0003] A general problem with audio reproduction using loudspeakers is that the reproduction is usually optimal only within one or a small range of listener positions. Worse, the quality of the audio reproduction changes significantly when the listener changes position or moves. When the listening position changes away from the optimal point, the induced spatial auditory image is unstable. The stereo image collapses into the nearest loudspeaker.

[0004] This problem has been addressed in previous literature, including [1], by tracking the listener position and adjusting gain and delay to compensate for deviations from the optimal listening position. [2] shows an extension of how to adapt to the spatial radiation characteristics of the speakers in use. Listener tracking is also used in conjunction with crosstalk cancellation (XTC), see for example [3]. XTC requires extremely precise positioning of the listener, so listener tracking is almost essential.

[0005] Previous listener-position adaptive gain compensation methods for loudspeaker signals assumed that the sound energy (and therefore the required compensation gain) has a constant roll-off with distance. For example, for an acoustic point source, the theoretical roll-off ("slope") of the sound energy at that distance is 6 dB per doubling of distance. Other slope values ​​can also be applied. In practice, however, these correlations only apply to very dry conditions (close to an anechoic chamber), which are rarely found in real-world sound reproduction environments.

[0006] Therefore, a concept involving a compensating gain scheme is desired which is also able to take into account a reproduction environment comprising a certain amount of reverberant sounds, with the aim of optimizing the quality of the output audio signal of a loudspeaker for listeners at different listening positions.

[0007] This object is achieved by the subject-matter of the independent claims.

[0008] Advantageous embodiments are the subject matter of the dependent claims. Summary of the invention

[0009] The object of the present invention is to provide a more realistic distance gain compensation, taking into account the fact that reverberant energy is present in a real reproduction environment (room / reproduction space). This difficulty can be overcome by taking into account information about the reverberant effect in the gain adjustment / compensation. In particular, a roll-off gain compensation function is used to map the listener-to-loudspeaker distance to a compensation gain, e.g. taking into account the effect of reverberation. The idea of ​​a basic embodiment of the present invention is that due to the presence of reverberation in the sound reproduction environment, the gain to be compensated does not increase uniformly with increasing listener-to-loudspeaker distance, i.e. increases by a fixed factor. This is based on the recognition that in a real room, the roll-off rate of the sound energy with increasing distance between the loudspeaker position and the listener is slower than in an anechoic reproduction environment. For example, due to reverberation, the attenuation of the sound energy may decrease with increasing distance from the listener to the loudspeaker. For example, the roll-off gain compensation function reflects this correlation, which takes into account that the roll-off compensated by the compensation gain becomes monotonically shallower with increasing listener-to-loudspeaker distance. Although using a roll-off gain compensation function in this way may appear to increase computational complexity compared to a gain adjustment that accounts for a constant roll-off of sound energy, in practice such gain adjustment increases the stability of the rendering and the accuracy with which the loudspeaker reproduces the sound at the listener's position.

[0010] Thus, embodiments relate to an audio processor for performing audio rendering by generating rendering parameters, the rendering parameters determining a speaker signal to be reproduced by a set of speakers from an audio signal. The audio processor is configured to obtain reverberation effect information and perform gain adjustment so as to determine a gain for generating a speaker signal for a speaker from the audio signal according to a listener position. The audio processor is configured to use a roll-off gain compensation function for at least one speaker in the gain adjustment according to the reverberation effect information, for mapping a listener-to-speaker distance of the at least one speaker to a listener-to-speaker distance compensation gain of the at least one speaker, the compensated roll-off being monotonically shallower as the listener-to-speaker distance increases. In other words, the roll-off gain compensation function can be configured to compensate for a roll-off of sound energy that is monotonically shallower as the listener-to-speaker distance increases, i.e., the roll-off of sound energy decreases as the listener-to-speaker distance increases. The slope of the roll-off gain compensation function can be monotonically shallower as the listener-to-speaker distance increases. For example, a "shallower" back-off gain increases more slowly when the listener-to-speaker distance is larger than when the listener-to-speaker distance is smaller, ie, the back-off gain increases at a smaller rate as the listener-to-speaker distance increases.

[0011] For example, the reverberation effect information may indicate the amount of reverberation effective in the reproduction room of the audio rendering, or may indicate whether the reverberation is effective in the reproduction room of the audio rendering. According to an embodiment, the reverberation effect information may include a first compensation roll-off slope of a roll-off gain compensation function, a second compensation roll-off slope of a roll-off gain compensation function, a near-field attenuation parameter, a far-field attenuation parameter, a critical distance parameter, and / or a near-field-far-field conversion parameter. The first compensation roll-off slope and the second compensation roll-off slope may indicate a compensation gain by distance or a compensation gain of sound energy by distance. The near-field attenuation parameter and the far-field attenuation parameter may indicate a roll-off of sound energy by distance, wherein the near-field attenuation parameter may indicate a higher attenuation than the far-field attenuation parameter. The first compensation roll-off slope may be related to the near-field attenuation parameter, and the second compensation roll-off slope may be related to the far-field attenuation parameter. The critical distance parameter may indicate a distance to a speaker in a group of speakers, such as a boundary distance, wherein the distance separates two distance zones associated with different reverberation effects. For example, a first distance zone (i.e., near field) having a distance less than a boundary distance may have a higher roll-off of sound energy than a second distance zone (i.e., far field) having a distance greater than the boundary distance. A critical distance parameter may indicate a distance to a speaker in a set of speakers at which the energy of the direct sound is equal to the energy of the reverberant sound. A near-field-far-field transition parameter may indicate how fast the transition between near-field attenuation and far-field attenuation is, for example, how the roll-off gain compensation function transitions from a first distance zone to a second distance zone.

[0012] The listener position may be defined by coordinates indicating the position of the listener within the reproduction space, such as the position of the listener's body, the position of the listener's head, or the position of the listener's ears, such as tracking data. For example, the listener's position may be described using Cartesian coordinates, spherical coordinates, or cylindrical coordinates. In addition to the listener's absolute position, the listener position may also represent the listener's relative position, for example relative to a reference speaker in a set of speakers or relative to each speaker in a set of speakers or relative to an optimal point in the reproduction space or relative to any other predetermined position in the reproduction space.

[0013] A further embodiment relates to a method for audio rendering by generating rendering parameters that determine loudspeaker signals to be reproduced by a set of loudspeakers from an audio signal. The method includes obtaining reverberation effect information and performing gain adjustment to determine gains for generating loudspeaker signals for the loudspeakers from the audio signal based on a listener position. Based on the reverberation effect information, the gain adjustment uses a roll-off gain compensation function for at least one loudspeaker to map a listener-to-loudspeaker distance of the at least one loudspeaker to a listener-to-loudspeaker distance compensation gain of the at least one loudspeaker, the compensated roll-off being monotonically shallower as the listener-to-loudspeaker distance increases.

[0014] A further embodiment relates to a computer program or a digital storage medium storing the computer program. The computer program comprises a program code for instructing the computer to perform one of the methods described herein, when the program is executed on a computer.

[0015] Further embodiments relate to a bitstream as described herein or a digital storage medium storing said bitstream.For example, the bitstream may include reverberation effect information and / or listener position and / or loudspeaker signals and or audio signals.

[0016] The method, computer program and bit stream described herein are based on the same considerations as the audio processor described herein. Incidentally, the method, computer program and bit stream may have all features and / or functions associated with an audio processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are not necessarily drawn to scale, with emphasis generally being placed on illustrating the principles of the invention. In the following description, various embodiments of the invention will be described with reference to the following drawings, in which:

[0018] Figure 1 A schematic diagram showing an embodiment of an audio processor determining gain and delay;

[0019] Figure 2 A schematic diagram showing an embodiment of amplitude translation;

[0020] Figure 3 A schematic diagram showing an embodiment of an audio processor configured for gain adjustment;

[0021] Figure 4 A schematic diagram showing compensation gain versus listener-to-speaker distance;

[0022] Figure 5 A Level 1 processing system is shown as an example of an audio processor described herein;

[0023] Figure 6 shows an example of a roll-off gain compensation function,

[0024] Figure 7 The code snippet of the initialization phase is exemplarily shown;

[0025] Figure 8 The following code snippet shows the release phase:

[0026] Fig. 9 The code snippet of the reset phase is exemplarily shown;

[0027] Figure 10a to 1 0i shows an exemplary code snippet of the real-time parameter update phase; and

[0028] Figures 11a to 11cThe code snippet of the audio processing stage is shown as an example. DETAILED DESCRIPTION

[0029] The same or equivalent elements or elements having the same or equivalent functions are denoted by the same or equivalent reference numerals in the following description even if they are shown in different drawings.

[0030] In the following description, a plurality of details are provided to more fully explain embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may also be implemented without these specific details. In other cases, in order to avoid obscuring embodiments of the present invention, well-known structures and devices are shown in block diagram form rather than in detail. In addition, unless otherwise specifically stated, the features of the different embodiments described below may be combined with each other.

[0031] Various examples are described below that help achieve more effective compression when using gain and / or delay adjustments controlled by listener position. For example, gain adjustment and / or delay adjustment can be added to other parameter adjustments for sound rendering, or can be provided separately.

[0032] In order to facilitate the understanding of the following examples of the present application, this description first introduces a possible device that can match the examples of the present application that are subsequently outlined. The following will first introduce an embodiment of an apparatus for generating speaker signals for multiple speakers. The following will outline more specific embodiments and explain how they can be applied individually or in groups. Figure 1 Details of the device.

[0033] Figure 1 The apparatus is generally indicated using reference numeral 10 and is used to generate speaker signals 12 for a plurality of speakers 14 in a manner so as to apply the speaker signals 12 on or to the plurality of speakers 14 to render at least one audio object at a predetermined virtual position.

[0034] The device 10 may be configured for a specific arrangement of speakers 14, i.e., a specific location where multiple speakers 14 are located or located and oriented. However, the device may also be configured for a different speaker arrangement of speakers 14. Likewise, the number of speakers 14 may be two or more, and the device may be designed for a fixed number of speakers 14, or may be configured to handle any number of speakers 14.

[0035] The device 10 includes an interface 16 at which the device 10 receives an audio signal 18 representing at least one audio object. For example, the device 10 can be configured to decode the audio signal 18 from a bitstream. For now, it is assumed that the audio input signal 18 is a mono audio signal that represents an audio object, such as the sound of a helicopter or the like. Other examples and more details will be provided below. Alternatively, the audio input signal 18 can be a stereo audio signal or a multi-channel audio signal. In any case, the audio signal 18 can represent the audio object in the time domain, the frequency domain, or any other domain, and can represent the audio object in a compressed or uncompressed manner.

[0036] like Figure 1 As shown, the device 10 also includes an object position input 20 for receiving an expected virtual position 21. That is, at the object position input 20, the device 10 is informed about the expected virtual position 21, and the audio object will be virtually rendered at the expected virtual position 21 by applying the speaker signal 12 at the speaker 14. That is, the device 10 receives information of the expected virtual position 21 at the input 20, which can be provided relative to the arrangement / position of the speaker 14, relative to the sweet spot, relative to the position and / or head direction of the listener and / or relative to the coordinates of the real world. The information can be based on a Cartesian coordinate system or a polar coordinate system. For example, it can be based on a room-centered coordinate system or a listener-centered coordinate system, which can be either a Cartesian coordinate system or a polar coordinate system.

[0037] Additionally, the apparatus 10 comprises a listener position input 30 for receiving an actual position of a listener. The listener position 31 may be defined by coordinates indicating a position of the listener position within the reproduction space, such as the position of the listener's body, the position of the head or the position of the ear, such as tracking data, i.e. information of the listener position over time. For example, the listener position 31 may be described using Cartesian coordinates, spherical coordinates or cylindrical coordinates. In addition to the absolute position of the listener, the listener position 31 may also represent a relative position of the listener, such as relative to a reference loudspeaker in a set of loudspeakers, or relative to an optimal point in the reproduction space, or relative to any other predetermined position in the reproduction space.

[0038] For example, where the virtual position 21 is intended to define the position of an audio object relative to a listener position 31 , the apparatus 10 may not necessarily require the listener position input 30 to receive the listener position 31 . This is because the virtual position 21 already takes the listener position 31 into account.

[0039] like Figure 1As shown, the apparatus 10 may include a gain determiner 40 configured to determine gains 41 for the plurality of loudspeakers 14 based on the expected virtual position 21 received at the input 20 and / or the listener position 31 received at the input 30. According to an embodiment, the gain determiner 40 may calculate an amplitude gain for each loudspeaker signal so as to shift the expected virtual position 21 between the plurality of loudspeakers 14 and / or so as to compensate for the roll-off of the sound energy. The gain 41 provided by the gain determiner 40 may represent a compensation gain, such as Figure 3 Alternatively, as outlined in more detail below, see Figure 2 , the panning gain g applied to each loudspeaker signal n Can include horizontal component and the vertical component For example, and optionally another component corresponding to the compensation gain, see Figure 3 The index n represents a positive integer in the range of 1≤n≤i, where i represents the number of loudspeakers 14. The gain determiner 40 may be configured to determine a respective gain 41 for each loudspeaker.

[0040] In addition, or alternatively, the device 10 may include a delay determiner / controller 50 to determine / control delays 51 for multiple speakers 14 based on the expected virtual position 21 received at the input 20 and / or the listener position 31 received at the input 30, and the delay determiner 50 may be configured to determine the respective delay 51 for each speaker so that when the speaker signal 12 is applied on or to the multiple speakers 14, at least one audio object is rendered at the expected virtual position, and / or the speaker signals reproduced by the speakers 14 arrive at the listener simultaneously.

[0041] The apparatus 10 may include an audio renderer 11 configured to render the audio signal 18 based on a gain 41 and / or a delay 51 in order to derive a speaker signal 12 from the audio signal 18 .

[0042] about Figure 2 , a possible 3D translation performed by the translation gain determiner 40 is described in more detail.

[0043] The loudspeakers 14 may be arranged in one or more horizontal layers 15. Figure 2As shown, the first group of speakers 141 to 145 of the plurality of speakers 14 can be arranged in a first horizontal layer 151, and the second group of speakers 146 to 148 of the plurality of speakers 14 can be arranged in a second horizontal layer 152. That is, the first group of speakers 141 to 145 are accurately arranged at a similar height, and the second group of speakers 146 to 148 are accurately arranged at a similar height. The first group of speakers 141 to 145 can be set at a first height or close to the first height, and the second group of speakers 146 to 148 can be arranged at a second height or close to the second height, for example, higher than the first height. According to Figure 2 In the illustrated embodiment, the listener position 31 is exemplarily arranged in the first horizontal layer 151 .

[0044] In the following, we will take a 3D rendered object as an example, where an object 1041 (e.g., a sound source) is panned to a direction (as seen from the listener 100) between two physically existing loudspeaker layers (at different heights). The object 1041 is amplitude-panned in the first layer 151 by passing the object signal to loudspeakers having different first layer level gains, for example, by passing the object signal to loudspeakers 141 to 145 so that it is amplitude-panned to the bottom layer, i.e., the first layer 151, see Figure 2 For example, in this horizontal panning, for each loudspeaker in the first group of loudspeakers 141 to 145, the horizontal component of the respective panning gain 41 is determined. Likewise, object 1041 is translated by amplitude in the second layer 152 to Figure 2 For example, in this horizontal panning, for each speaker in the second group of speakers 146 to 148, the horizontal component of the respective panning gain 41 is determined. It can be seen that the positions 104'1 and 104"1 may be selected so that they overlap each other in the vertical direction and / or so that the vertical projection of the expected position 1041 coincides with the positions 104'1 and 104"1. Figure 2 The rendering of the final object position 1041 by applying an amplitude translation between the layers 15 is shown, i.e., a vertical translation is shown. The virtual objects at the positions 104'1 and 104"1 are used as virtual speakers and are amplitude translated by the gain determiner 40 to render the virtual object at the desired position 1041 between the two layers 151 and 152. For example, in this vertical translation, a vertical component of the respective panning gain 41 is determined for each speaker in the first group of speakers 141 to 145 and the second group of speakers 146 to 148. The result of amplitude panning between layers 151 and 152 is two gain factors, the horizontal component of each speaker and the vertical component The loudspeaker signals are weighted by these two factors, e.g. so that the sound source of the audio signal is panned to the desired sound source position of the audio signal. The weighting of the horizontal panning between the (real) loudspeaker layers 15 may additionally be frequency dependent, to compensate for the effect of the vertical panning that different frequency ranges may be perceived at different heights.

[0045] In the following, the case of a 3D rendered object is described with respect to an example where an object 1042 is translated above or below the outermost layer. The direction or position 1042 of the object may not be within the direction range between the two layers 151 and 152, as discussed with respect to the object position 1041. For example, the expected position 1042 of the object is above or below the (actually existing) layer 15, which here refers to any available layer, in particular the lower layer, i.e., the first layer 151. For example, the direction / position 1042 of the object is below the bottom speaker layer, i.e., the first layer 151, which speaker arrangement has been used as Figure 2 . In this case, the panning gain determiner 40 applies a horizontal amplitude panning to the bottom layer to render the object 1042 in this layer 151, see the resulting position 104'2. The resulting position 104'2 may represent a virtual sound source position corresponding to the projection of the sound source position of the desired audio signal (see 1042) onto the nearest speaker layer (see 151). More generally, a 2D amplitude panning is applied between the speakers 141 to 145 belonging to the speaker layer closest to the object 1042 (i.e. the first layer 151). For example, in this horizontal panning, for each speaker in the first group of speakers 141 to 145, the horizontal component of the respective panning gain 41 is determined. Then, further amplitude panning is performed between the speakers 141 to 145 belonging to the nearest speaker layer, i.e., the first layer 151, and the audio signal is spectrally shaped so that the sound is presented by the speakers 141 to 145 of the nearest speaker layer (i.e., the first layer 151), which simulates the sound from another virtual sound source position 104"2 that is offset from the nearest speaker layer (i.e., the first layer 151), towards the sound source position of the desired audio signal, see 1042. Since there are no real speakers in the vertical top or bottom directions, the vertical signal at 104"2 can be equalized to simulate the coloration of the top or bottom sound, respectively. The vertical signal is then sent to the speakers designated for the top / bottom directions. In order to render the final object position 1042, the panning gain determiner 40 may be configured to further apply an amplitude panning between the virtual sound source position 104'2 and the further virtual sound source position 104"2, thereby determining a second panning gain for panning between the virtual sound source position 104'2 and the further virtual sound source position 104"2, so that the speakers 141 to 145 of the nearest speaker layer render the audio signal from the sound source position 1042 of the desired audio signal. If the sound source position 1042 of the desired audio signal is located below one or more speaker layers, i.e. below the first layer 151, spectral shaping of the audio signal may be performed using a first equalization function that simulates the timbre of bottom sounds, and / or if the sound source position of the desired audio signal is located above one or more speaker layers, i.e. above the second layer 152, spectral shaping of the audio signal may be performed using a second equalization function that simulates the timbre of top sounds.

[0046] Figure 3 An embodiment of an audio processor 10 is shown for performing audio rendering by generating rendering parameters 100 , see audio renderer 11 , which determine deriving a loudspeaker signal 12 from an audio signal 18 to be reproduced by a set of loudspeakers 14 . Figure 3 The embodiment shown focuses on the gain determiner 40. Optionally, it can be combined with Figure 1 The delay determiner 50 is combined. Figure 3 The illustrated embodiment provides details regarding the use of gain determiner 40 to determine compensation gains 41. These gains may also be expressed as Figure 1 The gain provided by the gain determiner shown in FIG. Figure 1 As described above, each compensation gain 41 may represent a respective component of a respective gain to be applied to a respective speaker.

[0047] The gain determiner 40 is configured to perform gain adjustment so as to determine a gain 41 for generating a loudspeaker signal 12 for the loudspeaker 14 from the audio signal 18 based on the listener position 31. For example, the gain adjustment is performed according to a gain associated with anechoic environment conditions so as to take into account the influence of reverberation. Thus, the gain 41 determined by the gain determiner 40 is more suitable for a real-world sound reproduction environment.

[0048] like Figure 3 As shown, the gain determiner 40 of the audio processor 10 obtains the reverberation effect information 110. The reverberation effect information 110 may indicate whether the reverberation in the reproduction space 112 is effective and / or the reverberation condition in the reproduction space 112. The audio processor 10 may be configured to obtain the reverberation effect information 110 from a bitstream or from side information of a bitstream.

[0049] The audio processor 10 is configured to use a roll-off gain compensation function 42 for the at least one speaker 14 in the gain adjustment according to the reverberation effect information 110, for mapping a listener-to-speaker distance 44 of the at least one speaker 14 to a listener-to-speaker-distance compensation gain 46 of the at least one speaker 14, wherein for the roll-off gain compensation function, the compensation roll-off becomes monotonically shallower as the listener-to-speaker distance 44 increases, see also Figure 4 and Figure 6 For example, the audio processor 10 is configured to determine, adjust or select a roll-off gain compensation function 42 based on the reverberation effect information 110. The listener-to-speaker distance compensation gain 46 determined for at least one speaker 14 may represent a gain 41 provided to the audio renderer 11 for deriving the respective speaker signals 12 to be reproduced by the respective speakers 14 from the audio signal 18.

[0050] For at least one speaker 14 using gain adjustment, the listener position 31 may indicate a listener-to-speaker distance 44. Alternatively, the listener position 31 may include a listener-to-speaker distance 44 for each speaker 14 in a group of speakers 14. Alternatively, the listener position 31 may also represent an absolute position of the listener 1 in the reproduction space 112. In this case, the audio processor 10 may be configured to additionally obtain information about the position of at least one speaker 14 using gain adjustment or the positions of all speakers 14 within the reproduction space 112. The audio processor 10 may be configured to determine the respective listener-to-speaker distances for at least one speaker 14 using gain adjustment based on the listener position 31 and the positions of the respective speakers 14.

[0051] The roll-off gain compensation function 42 used by the gain determiner 40 will be Figure 4 (See 42 β1 and 42 β2 )and Figure 6Explain in more detail.

[0052] Figure 4 Schematic diagram showing two different roll-off gain compensation functions for the near-field-to-far-field transition parameter beta, see 42 β1 and 42 β2 The larger beta is, the faster the transition between near-field decay and far-field decay. The near-field-far-field transition parameter beta may be composed of the reverberation effect information 110 discussed herein.

[0053] For the same critical distance 44 12 , for example, the distance to the relevant loudspeaker (ie, the loudspeaker to which the roll-off gain compensation function may be applied) is 4 meters, and the roll-off gain compensation function 42 is described as an example. β1 and 42 β2 . Critical distance 44 12 The roll-off gain compensation function can be shifted along the listener-to-loudspeaker-distance axis, see 44. The greater the effective amount of reverberation in the reproduction space, the greater the critical distance 44. 12 The smaller. Figure 6 The critical distance 44 is shown as an example 12 The roll-off gain compensation function at two meters. Critical distance 44 12 May be composed of reverberation effect information 110 discussed herein. Critical distance 44 12 It can represent the distance at which the energy of the direct sound is equal to the energy of the reverberant sound.

[0054] In addition, the near-field roll-off gain compensation function is described 43 nf and far-field roll-off gain compensation function 43 ff . Figure 4 The relationship of the compensation gain 46 to the listener-to-loudspeaker distance 44 is shown.

[0055] The reverberation effect information 110 may indicate that the sound decays more slowly as the distance from the speaker 14 increases. For example, near each speaker 14, i.e., in the near field (see 441), the sound energy rolls off faster than far away from each speaker 14, i.e., in the far field (see 442). The reverberation effect information 110 may include near field decay parameters and far field decay parameters, e.g., see Fig.10c and decay_1_dB and decay_2_dB in 10i. Near-field roll-off gain compensation function 43 nf The compensation gain 46 represents the compensation gain of the roll-off (i.e., the roll-off of the sound energy) according to the near-field attenuation parameter, and the far-field roll-off gain compensation function 43 ff = represents the compensation gain 46 for compensating the roll-off (ie, the roll-off of the sound energy) according to the far-field attenuation parameter. In determining the roll-off gain compensation function (see 42 β1 and 42 β2), consider the near-field attenuation parameters and far-field attenuation parameters. Roll-off gain compensation function, see 42 β1 and 42 β2 An overall compensation gain 46 is schematically shown for compensating for the sound energy roll-off over the near-field and far-field listener-to-speaker distance 44. Figure 4 As shown, the (for example, total) roll-off gain compensation function is shown in 42 β1 and 42 β2 , transitioning between near-field attenuation and far-field attenuation.

[0056] Roll-off gain compensation function (see 42 β1 and 42 β2 ) represents a listener-to-speaker distance compensation gain 46, which is applied to the speaker signal 12 to compensate for reverberation associated with sound energy roll-off. Figure 4 As shown, the roll-off gain compensation function (see 42 β1 and 42 β2 ) is configured so that the listener-to-speaker distance compensation gain 46 increases slowly as the listener-to-speaker distance 44 increases, that is, the roll-off gain compensation function 42 becomes monotonically shallower as the listener-to-speaker distance 44 increases, for example, the change in the compensation gain per unit distance decreases as the listener-to-speaker distance 44 increases.

[0057] For example, the roll-off gain compensation function (see 42 β1 and 42 β2 ) has a first slope (see 42′) in a first distance zone 441 (eg, in the near field) β1 and 42' β2 ) (e.g., a first compensating roll-off slope), and having a second slope (see 42″) in a second distance region 442 (e.g., in the far field) β1 and 42” β2 ), (e.g., a second compensation roll-off slope), wherein the first slope 421 is greater than the second slope 422, and the distance involved in the first distance zone 441 is less than the distance involved in the second distance zone 442. The first slope 421 and / or the second slope 422 may be indicated by the reverberation effect information 110. The reverberation effect information 110 may further indicate a boundary distance separating the first distance zone 441 and the second distance zone 442, such as a critical distance 44 12 .Border distance 44 12 It may correspond to the distance from the loudspeaker 14 at which the energy of the direct sound is equal to the energy of the reverberant sound in the reproduction space 112 .

[0058] According to an embodiment, the reverberation effect information 110 may indicate how the roll-off gain compensation function 42 transitions from the first distance zone 441 to the second distance zone 442 , for example using a near-field-to-far-field transition parameter beta. Figure 4 The roll-off gain compensation function 42 is shown as an example β1 , with roll-off gain compensation function 42 β2 Compared to the roll-off gain compensation function 42 β1 By taking into account the specific transition of the reproduction space 112 between the near-field sound energy attenuation and the far-field sound energy attenuation, the accuracy of determining the compensation gain 41 can be improved.

[0059] The audio processor 10 is configured to perform gain adjustments so that the listener position 31 becomes the best point in acoustic or perceptual terms relative to the set of loudspeakers 14, i.e. the listener 1 can perceive the sound intended by the mixer reproduced by the loudspeaker set 14. By means of a specific gain adjustment, artifacts that the listener 1 may perceive at his position can be reduced.

[0060] The following will be combined Figure 3 and Figure 4 The relationship between the reverberation effect information 110 and the gain adjustment using the roll-off gain compensation function 42 is described in more detail.

[0061] The reverberation effect information 110 may indicate the amount of reverberation effective in the reproduction room (i.e., the reproduction space 112), i.e., indicating how much sound or signal is reflected in the reproduction space 112, for example, from walls or furniture. The amount of reverberation effective in the reproduction space 112 may indicate how much reflections accumulate and then decay as the sound is absorbed, for example, by the surface of an object / wall in the reproduction space 112. In this case, the audio processor 10 may be configured to select a roll-off gain compensation function, see Figure 3 42 and Figure 4 of 42 β1 and 42 β2 (hereinafter generally referred to by reference number 42), or adjusting the roll-off gain compensation function 42 to obtain a roll-off gain compensation function 42, wherein the intensity of the compensating roll-off becomes monotonically shallower as the listener-to-speaker distance 44 increases, and the greater the effective reverberation amount in the reproduction space 112. Figure 4 The roll-off gain compensation function 42 is shown as an example. β2 The roll-off gain compensation function 42 of the reproduction space 112 having a larger effective amount of reverberation than the related reproduction space 112 β1 Roll-off gain compensation function 42 β1 The roll-off gain compensation function 42 may be used to compensate for the roll-off of the sound energy in the reproduction space 112, where the sound energy does not roll off or lose as quickly as in a reproduction space 112 with less effective reverberation. β2Therefore, the audio processor 10 should adjust or select the roll-off gain compensation function so that the listener-to-speaker distance compensation gain 46 starts to increase slowly as the listener-to-speaker distance 44 increases, and the gain (see critical distance 44) increases. 12 ), the greater the amount of effective reverberation in the reproduction space 112. This is based on the recognition that an increase in the amount of effective reverberation in the reproduction space reduces the roll-off of the sound energy. Adjustment of the roll-off gain compensation function 42 allows for increased accuracy in determining the compensation gain 41.

[0062] The reverberation effect information 110 may indicate whether the reverberation is effective in the reproduction space 112. Only when the reverberation is effective in the reproduction space 112, the roll-off gain compensation function 42 described herein can be used, which becomes shallower / slower monotonically with increasing listener-to-speaker distance 44. If the reverberation effect information 110 indicates that the reverberation is not effective in the reproduction space 112, the audio processor 10 may be configured to use a further roll-off gain compensation function whose compensated roll-off is constant, for example, in this case the near-field roll-off gain compensation function 43 may be used. nf . For example, a further roll-off gain compensation function may be configured to compensate for a predefined sound energy, for example a roll-off of, for example, 6 dB per doubling of the listener-to-loudspeaker distance 44. Reverberation may cause a different attenuation of sound energy in the near field of the loudspeaker 14 compared to the far field of the loudspeaker. However, if there is no effective reverberation in the reproduction space 112, such near-field and far-field differences do not need to be taken into account. Therefore, for this case, a simpler determination of the compensation gain may be performed. This enables the compensation gains for different reproduction spaces 112 to be determined efficiently and with reduced complexity.

[0063] The concept of the basic embodiment of the present invention is described below. In particular, the distance gain compensation, see Figure 3 The roll-off gain compensation function 42 and Figure 4 of 42 β1 and 42 β2 , taking into account the fact that there is reverberant energy in the reproduction room 112, so as the distance between the speaker position and the listener 1 increases, the sound energy rolls off more slowly. That is, Gain / level adjustment is considered This is done by taking into account information on the amount of reverberation present in the reproduction room 112. For example, for a point source, the theoretical roll-off ("slope") of acoustic energy with distance is 6 dB per doubling of distance. Taking into account room reverberation, User adaptive speaker rendering gain compensation strength (slope) becomes shallower (less steeper) with distance, see Figure 4 and Figure 6 . One parameter that defines this roll-off variation can be related to the so-called “critical distance” or boundary distance44 12In acoustics, the boundary distance refers to the distance at which the energy of the direct sound is equal to the energy of the reverberant sound [4]. For the user adaptive speaker rendering scheme, the critical distance 44 12 The relevant control parameters are very effective in controlling the appropriate compensation characteristics.

[0064] Therefore, according to an embodiment, the result of the above idea is that the audio signal processor 10

[0065] Wherein the value of the gain compensation slope of at least one loudspeaker signal depends on the position of the listener / the distance between the listener and the loudspeaker 44

[0066] Optionally, the delay may also be adjusted based on the reverberation data of the reproduction environment room 112.

[0067] The slope of the larger distance is smaller (shallower) than the slope of the smaller distance

[0068] There are at least two distance zones 441 and 442 to which different slope values ​​or slope value ranges are applied, and the slope value of the nearby (first) zone 441 is greater than the slope value of the distant (second) zone 442 .

[0069] ο Among them, the "critical distance" 44 12 The relevant parameters are used to define the boundary between the near (first) zone and the far (second) zone.

[0070] o Wherein the slope value of the near (first) zone 441 is steeper than the slope value of the far (second) zone 442.

[0071] o Wherein the slope parameters of the near (first) zone 441 are used / accepted and applied to this zone

[0072] o Wherein the slope parameters of the far (second) zone 442 are used / accepted and applied to this zone

[0073] Optionally, a transition parameter is defined for determining the transition (eg, roundness) between the two regions 441 and 442 and applied to the roll-off gain compensation function 42.

[0074] According to an embodiment of the present invention, the audio processor 10 is configured to generate a set of one or more parameters (for example, parameters that can affect the delay, level or frequency response of one or more audio signals, such as rendering parameters 100) for each speaker in a group of one or more speakers 14. This time, the parameters are determined based on a listener position 31 (the listener position 31 can be the position of the entire body of the listener 1 in the same room (i.e., the reproduction space 112) as the group of one or more speakers 14, or only the head position of the listener 1, or, for example, the ear position of the listener 1. The listener position 31 can be, for example, a position of a reference group of one or more speakers 14, such as the distance between the listener's head and the group of one or more speakers 14) and the speaker position of the group of one or more speakers 14, and the speaker signal 12 reproduced by each speaker 14 is derived from the audio signal 18. The audio processor 10 is configured to generate a set of one or more parameters for a group of one or more speakers 14 based on information about the reverberation characteristics of the reproduction environment (room) (i.e., information of the reverberation effect information 110). Specifically, the calculation of the level (gain 41 ) value of the speaker signal 12 is based on information on the level of the reverberant sound present in the reproduction room 112 .

[0075] Taking into account this information about the level of reverberant sound, the present invention improves the user adaptive loudspeaker rendering result by utilizing a strength (slope) of the level (gain 41) compensation that becomes shallower (gradualer) with increasing distance (i.e., listener-to-loudspeaker distance 44). An important parameter defining the distance-dependent slope variation may be related to the so-called "critical distance", see 44 12 The term “critical distance”44 12 In acoustics, it is referred to as the distance at which the energy of the direct sound radiated from the sound source is equal to the energy of the reverberant sound [4]. For the user-adaptive speaker rendering scheme of the present invention, it is found that the critical distance 44 12 The relevant control parameters are very effective in controlling the appropriate compensation characteristics. In addition, a distance significantly below the critical distance 44 can be defined and used. 12 The slope value at the listener position 31 and clearly exceeds the critical distance 44 12 The slope value at the listener position 31.

[0076] This can be implemented with the audio processor 10. For example, the audio processor 10 obtains information about the listener positioning (i.e., the listener position 31), the speaker positioning (i.e., the speaker position) and the reverberation characteristics of the reproduction room (i.e., the reverberation effect information 110), such as the critical distance of the room, the nearby slope parameter (e.g., indicating the first slope 421) or the distant slope parameter (e.g., indicating the second slope 422). The audio processor 10 can calculate a set of one or more parameters based on this information. Using a set of one or more parameters, the input audio, alternatively the input audio signal 18, can be modified. By modifying the audio signal 18, the listener 1 receives an optimized audio signal at his position. With this optimized signal, the listener 1 can, for example, obtain an auditory experience at his position that is almost or completely the same as the listener's intended listening position. The ideal listener position is, for example, a position where the listener experiences the best audio perception without any modification of the audio signal, such as a sweet spot. This means, for example, that the listener 1 can perceive the audio scene at this position in the way expected at the production site. The ideal listener position can correspond to a position that is equidistant from all the speakers 14 (one or more speakers 14) used for reproduction.

[0077] Thus, the audio processor 10 according to the invention allows the listener 1 to change his / her position to different listener positions 31 and in each position, at least in some positions, have the same or at least partially the same listening sensation as the listener in the ideal listening position.

[0078] In summary, it should be noted that the audio processor 10 is capable of adjusting at least one of the delay, level or frequency response of one or more audio signals 18 depending on the listener positioning, the loudspeaker positioning and / or the loudspeaker characteristics, with the aim of achieving an optimized audio reproduction for at least one listener 1. The level is also adjusted depending on the information about the reverberation characteristics 110 of the reproduction room 112.

[0079] Now, an embodiment of the invention is described for adaptive loudspeaker rendering.

[0080] First a general description should be made. As an alternative to rendering and binauralizing the MPEG-I scene to headphones, playback through loudspeakers can be specified. In this mode of operation the MPEG-I spatializer (HRTF-based renderer) is replaced by a dedicated loudspeaker-based renderer, as described below.

[0081] To achieve a high-quality listening experience, the loudspeaker setup assumes that the listener 1 is located at a dedicated fixed position, the so-called sweep point. Typically, in a 6DOF playback situation, the listener 1 is moving. Therefore, the 3D spatial rendering must instantly and continuously adapt to the changing listener position 31. This can be achieved using two hierarchically nested technical levels.

[0082] 1. For example, a gain 41 and a delay 51 are applied to the loudspeaker signal 12 so that the loudspeaker signal 12 arrives at the listener position 31 with similar gain and delay, i.e. also at the sweet spot. Optionally, a high-shelf compensation filter is applied to each loudspeaker signal 12 that is dependent on the current listener position 31 and the direction of the loudspeaker relative to the listener 1. In this way, when the listener 1 moves to an off-axis position of the loudspeaker 14 or further away, the high-frequency losses caused by the radiated high-frequency mode of the loudspeaker are compensated.

[0083] 2. Due to the 6DoF motion, the angles between the loudspeaker 14, the object, and the listener 1 change as the listener position 31 changes. Therefore, the 3D amplitude translation algorithm (e.g. Figure 2 The LSDF is updated in real time according to the relative position and angle of the changing listener position 31 and the fixed speaker configuration set in the LSDF. All coordinates (listener position 31, source position) can be converted in the listening room coordinate system, that is, converted into the coordinate system of the reproduction space 112.

[0084] Physical compensation level (level 1)

[0085] Figure 5 An embodiment of a Level 1 system 10 is shown with an overview of its main components and parameters. Figures 1 to 4 The audio processor 10 may include Figure 5 The features and / or functions described in the embodiments.

[0086] Level 1 : Real-time update compensation of speaker (frequency-dependent) gain and delay, see Audio Renderer 11, to implement " Enhanced content rendering . By utilizing tracked user position information, such as a version of the listener position 31, the listener 1 (i.e., user) can move within a large "sweet spot" (rather than a sweet spot), and experience a stable sound field within the large area, such as when listening to traditional content (e.g., stereo, 5.1, 7.1+4H). For immersive formats (i.e., non-stereo), when moving away from the sweet spot, the sound seems to separate from the speaker 14 instead of collapsing into the nearest speaker 14, i.e., the sound quality is somewhat close to the effect of wave field synthesis, but is limited to a single user experience. For stereo reproduction, the technology can provide stability of the left and right sound fields within a large range of the user position 31 (i.e., the range between the left and right speakers at any distance).

[0087] For example, level 1 Gain CompensationIt is based on the law of amplitude attenuation. In the free field, the amplitude is proportional to 1 / r, where r is the distance from the listener 1 to the speaker 14 (1 / r corresponds to a 6dB attenuation for every doubling of the distance). In the room 112, due to the presence of acoustic reflections and reverberation, the sound decays more slowly as the distance from the speaker 14 increases. Therefore, near-field attenuation, far-field attenuation and / or critical distance parameters, such as those composed of the reverberation effect information 110, can be used to specify the functional relationship between the attenuation rate and the distance from the speaker 14. In addition, there may be a near-field-to-far-field transition parameter beta, such as those composed of the reverberation effect information 110. The larger beta is, the faster the transition between the near-field attenuation and the far-field attenuation. Figure 6 An example of gain compensation as a distance function is shown, namely a roll-off gain compensation function 42 which is available to the gain determiner 40. In a reverberant field, the gain variation is smaller than in a free field.

[0088] For example, in level 1 Delay compensation The propagation delay from each speaker 14 to the listener position 31 is calculated and then a delay is applied to each speaker 14 to compensate for the propagation delay differences between the speakers 14. The delays may be normalized (an offset is added or subtracted) so that the minimum delay applied to the speaker signal 12 is zero.

[0089] Object Rendering Level (Level 2)

[0090] Level 2: User Tracked Object Translation can render point sources (objects, channels) within the 6DoF playback space, requiring Level 1 as a prerequisite. Therefore, it addresses the use case of "6DoF VR / AR rendering". Level 1 systems 10 may also include the following features and / or functions.

[0091] The 3D amplitude panning algorithm can be used for speaker layers, such as horizontal layers and height layers, such as Figure 2 Each layer can apply a 2D translation algorithm to project the object onto that layer. The final 3D object is rendered by applying an amplitude translation between the two virtual objects of the 2D translation in the two layers.

[0092] When the object is above the top layer, a 2D pan is applied to that layer. The final 3D object is rendered by applying an amplitude pan between the 2D panned virtual object and the (non-existent) object in the vertical direction. The vertical object's signal can be equalized to mimic the timbre of the top sound and evenly distributed to the top layer's speakers.

[0093] When an object is below the bottom layer, a 2D pan is applied to that layer. The final 3D object is rendered by applying an amplitude pan between the 2D panned virtual object and the (non-existent) object below it in the vertical direction. The vertical object's signal can be equalized to mimic the timbre of the bottom sound and evenly distributed to the bottom layer's speakers.

[0094] The vertical panning described is equally applicable to single-layer (eg 5.1) and multi-layer (eg 7.4.6) speaker setups.

[0095] Level 1 and Level 2 applied to object rendering render MPEG-I scenes faithfully as if they were rendered through headphones. This is a significant benefit compared to speaker-rendered MPEG-I content without applying adaptive tracking (1 and 2).

[0096] Physical compensation level (level 1)

[0097] The following code snippets describe an embodiment of gain and delay adjustment based on listener position, see Figure 10c to 1 0i and Fig.11b and Fig.11c . Figure 1 Audio processor 10 or Figure 5 The Level 1 system 10 may provide the features and / or functionality described below with respect to gain and / or delay adjustment. Figure 3 The audio processor 10 may also include the features and / or functions described below with respect to delay adjustment. Optionally, Figure 3 The audio processor 10 may include the following features and / or functions regarding gain and / or delay adjustment. Optionally, Figure 1 Audio processor 10, Figure 3 The audio processor 10 and Figure 3 The audio processor 10 may include further features and / or functionalities as described below.

[0098] Data Elements and Variables

[0099] Below (see Figures 7 to 11c ) gives definitions and / or explanations of the data elements and variables used:

[0100] SFREQ_MIN minimum sampling rate [Hz] = 44100

[0101] SFREQ_MAX minimum sampling rate [Hz] = 48000

[0102] VSOUND Speed ​​of sound in air [m / s] = 340.0

[0103] MAX_DELAY Maximum delay [samples] = 960

[0104] OVERHEAD_GAIN overhead [lin] = 0.25

[0105] framesize The number of samples per frame, default: 256

[0106] sfreq_Hz sampling frequency of input audio, default: 48000

[0107] nchan Number of channels (speakers)

[0108] max_delay maximum delay [samples], default: MAX_DELAY

[0109] bypass_on 0: normal operation, 1: bypass, default: 0

[0110] ref_proc 0: normal operation, 1: similar to the best point processing, default: 0

[0111] cal_system 0: normal operation, 1: calibration system, default: 0

[0112] gain_on 0: gain off, 1: gain on, default: 1

[0113] delay_on 0: delay off, 1: on, default: 1

[0114] decay_1_dB Near-field sound attenuation per doubling of distance [dB], default: 8

[0115] decay_2_dB Double the far field sound attenuation per distance [dB], default: 0

[0116] beta 1: default near-field-far-field transition, >1 faster transition

[0117] crit_dist_m critical distance [m], default: 4

[0118] max_m_s Maximum moving speed [v, unit m / s], default: 1

[0119] max_m_s_s Maximum moving acceleration [a, unit m / s], default: 1

[0120] gain_ms Gain smoothing time constant [ms], default: 40

[0121] sweet_spot sweet spot position [m,m,m]

[0122] spk_pos Speaker coordinates [m,m,m]

[0123] listener_pos listener coordinates [m,m,m]

[0124] For example, all coordinates are relative to the listening room defined in the LSDF file. These parameters can be stored in the following structure.

[0125] Common data structures

[0126]

[0127] For example, the internal parameters calculated based on the above parameters and states are stored in the following structure: Internal data structure

[0128]

[0129]

[0130] Phase Description

[0131] The following describes an embodiment of gain and delay adjustment based on listener position using code snippets related to different stages. The embodiment may include an initialization stage (see Figure 7 ), release phase (see Figure 8 ), reset phase (see Fig. 9 ), real-time parameter update phase (see Figure 10a to 1 0i) and audio processing stage (see Figures 11a to 11c ). Figure 1 Audio processor 10, Figure 5 Level 1 System 10 and Figure 3 The audio processor 10 may include features and / or functions described for one or more stages, or individual features and / or functions for one or more stages.

[0132] initialization

[0133] Figure 7 The code snippet of the initialization phase is shown as an example.

[0134] Speaker setups can be loaded from LSDF files.

[0135] Initializes a structure of type rendering_gd_cfg_t with default values, with the nchan field set to the number of speakers in the speaker setup.

[0136] Initializes a structure of type rendering_gd_rt_cfg_t with default values. The loudspeaker position from the LSDF file is stored in the spk_pos field. If a ReferencePoint element is given in the LSDF file, its coordinates are stored in the sweet_spot field. If a calibration attribute is present, the field cal_system is set to the value of the calibration attribute.

[0137] The above structure is passed to the rendering_gd_init function.

[0138] release

[0139] Figure 8 The code snippet of the release phase is exemplarily shown.

[0140] Reset

[0141] Fig. 9 The code snippet of the reset phase is exemplarily shown. Fig. 9 Shows that all internal buffers are flushed.

[0142] Real-time update parameters

[0143] In the update thread, the virtual listener position is converted to the listening room coordinate system. This only applies to VR scenes. In AR scenes, the two coordinate systems coincide.

[0144] All further processing happens in the audio thread.

[0145] Update the structure of type rendering_gd_rt_cfg_t by setting the listener_pos field to the listener position (in the listening room coordinate system), see Fig.10a This structure is then passed to the rendering_gd_updatecfg function, see Fig.10a .

[0146] Calculate the compensation gain and delay of each speaker. The reference distance r_ref (at Fig.10a ) is the distance (dB, sample) at which the gain and delay compensation are zero. The gain and delay compensation are calculated based on the distance r from the speaker to the listener and the reference distance r_ref. The listener-to-speaker distance 44 is calculated based on the listener position 31 and the respective speaker positions 32, as Fig.10b The listener-to-loudspeaker distance 44 may represent a version of the listener position 31 .

[0147] In a free field, the sound decays by 6 dB for each doubling of the distance. In a room, the decay can be approximated by a smaller decay, for example 4 dB for each doubling of the distance. Alternatively, a critical distance (radius of the hall) can be considered. When approaching the loudspeaker, the decay is decay_dB for each doubling of the distance. Beyond the critical distance crit_dist_m, the sound decays only slowly. It is recommended to use the roll-off gain compensation function 42 (see Figure 6 , Fig.10c and FIG. 10i) to determine a gain compensation to compensate for the gain change caused by the sound attenuation.

[0148] Gain compensation can be based on the law of amplitude attenuation. In the free field, the amplitude is proportional to 1 / r, where r is the distance from the listener to the speaker (1 / r corresponds to a 6dB attenuation for every doubling of the distance). In the room, due to the presence of acoustic reflections and reverberation, the attenuation of sound slows down as the distance from the speaker increases. Therefore, near-field attenuation, far-field attenuation, and critical distance parameters can be used to specify the functional relationship between the attenuation rate and the speaker distance. In addition, there is a near-field-far-field conversion parameter beta47. The larger beta is, the faster the transition between near-field attenuation and far-field attenuation. The roll-off gain compensation function 42 may depend on the near-field-far-field conversion parameter beta 47. The near-field-far-field conversion parameter beta 47 can define how fast the roll-off gain compensation function 42 converts between the near field and the far field, that is, how fast the roll-off gain compensation function 42 converts from a steep increase in gain compensated by the listener-to-speaker distance 44 to a shallow increase or slight increase in gain compensated by the listener-to-speaker distance 44.

[0149] It is noted that the compensated roll-off becomes monotonically shallower as the listener-to-speaker distance 44 increases, which can be reflected by the slope of the compensated roll-off energy, which, when measured in the logarithmic domain, decreases monotonically with increasing listener-to-speaker distance 44.

[0150] The roll-off gain compensation function 42 maps a listener-to-speaker distance 44 associated with a loudspeaker to a listener-to-speaker distance compensation gain 41 of the loudspeaker associated with the listener-to-speaker distance 44. The roll-off gain compensation function 42 may be configured to compensate for a roll-off that becomes monotonically shallower as the listener-to-speaker distance 44 increases. As described above, in a reverberant reproduction space, the attenuation of sound energy in the near field and the far field may be different. Therefore, it is recommended to use a first attenuation parameter 481 (see decay_1_dB) for the near field (i.e., the first distance zone) and a second attenuation parameter 482 (see decay_2_dB) for the far field (i.e., the second distance zone), wherein the first distance zone is associated with a smaller listener-to-speaker distance 44 than the second distance zone. From Fig.10c As can be seen from FIG. 10i , the roll-off gain compensation function 42 takes into account the different attenuations 481 and 482 of the near field and the far field when determining the compensation gain 47 for a certain listener-to-speaker distance 44. For example, the roll-off gain compensation function 42 can take into account the degree of sound energy attenuation at the listener-to-speaker distance 44 according to a first attenuation parameter 481 (see pow_nf) and according to a second attenuation parameter 482 (see pow_ff). There is a critical distance 44 between the near field and the far field. 12 The acoustic energy attenuated according to the second attenuation parameter 482 (see pow_ff) may be scaled so that the acoustic energy attenuated according to the first attenuation parameter 481 and the second attenuation parameter 482 is at the critical distance 44.12 The first attenuation parameter 481 may represent a faster attenuation of the acoustic energy than the second attenuation parameter 482. Therefore, for the roll-off gain compensation function 42, the compensated roll-off becomes monotonically shallower as the listener-to-loudspeaker distance 44 increases.

[0151] Furthermore, the roll-off gain compensation function 42 may take into account how much sound energy is attenuated at the optimal point, see pow_ref at the optimal point r_ref. Thus, gain adjustment is performed so that the listener's position becomes the optimal point in an acoustic or perceptual sense relative to a set of loudspeakers. Considering the first attenuation parameter 481 and the second attenuation parameter 482, the sound energy attenuated at the optimal point may be determined.

[0152] Depending on the distance 44 between the loudspeaker and the listener position, the sound propagation time varies. These variations can be compensated by applying a delay. For example, an offset of MAX_DELAY / 2 is added to the compensation delay so that it is always positive, see Fig.10d Furthermore, the listener-to-loudspeaker distance may be taken into account when determining / adjusting the delays, as well as the distance between the sweet spot and the loudspeakers, see r_ref. Thus, the delay processing is performed so that the listener position is the sweet spot in an acoustic or perceptual sense relative to the set of loudspeakers.

[0153] Fig.10d It is shown that for each loudspeaker, a distance 44 from the listener position to the position of each loudspeaker can be determined, and based on the distance 44, a delay of each loudspeaker can be determined, see delay0[i].

[0154] from Fig.10d As can be seen, for each loudspeaker, an individual delay is determined, for example an absolute delay, see index i of delay variable delay0. Alternatively, the delay process may determine a reference loudspeaker in a set of loudspeakers and determine the delays of the other loudspeakers except the reference loudspeaker relative to the delay determined for the reference loudspeaker.

[0155] The overhead determined by OVERHEAD_GAIN can be used, see Fig.10e That is, when the listener is far away from the loudspeaker, the system can amplify the signal by up to 1 / OVERHEAD_GAIN. If the gain exceeds this value, the gain of all channels is scaled by the same factor so that the maximum gain is 1.0 (0dB). This is equivalent to the action of the inter-channel ganged limiter.

[0156] In addition to gain adjustments, delay adjustments can also be performed to reduce artifacts in the audio presentation due to delay variations.

[0157] According to an embodiment, the control of the delay processing can be performed by tailoring the listener's speed or by tailoring the delay, wherein the tailoring of the delay and the listener's speed can be controlled based on the maximum allowed listener speed (see max_m_s). For example, a maximum speed can be defined at which delay changes due to too fast a change in the listener's position will hardly produce artifacts in the audio rendering. Fig.10f The maximum delay variation (see delay_delta) determined based on the maximum allowed listener velocity is shown. The number of samples that the delay is allowed to vary from one frame to another is calculated as a function of the maximum allowed movement velocity max_m_s. The maximum allowed movement velocity max_m_s can be related to the maximum delay variation rate [v, in m / s].

[0158] According to alternative embodiments, the control of the delay processing may be performed by clipping the listener's acceleration or by clipping the time rate of change of the delay, wherein the time rate of change of the delay and the clipping of the listener's acceleration may be controlled based on a maximum allowed listener acceleration (see max_m_s_s). For example, a maximum acceleration may be defined for which delay changes due to too fast a change in the listener's position will hardly produce artifacts in the audio rendering. Figure 10g The maximum time rate of change of the delay determined based on the maximum allowed listener acceleration is shown, see delay_delta2. The number of samples by which the delay is allowed to change from one frame to another is calculated as a function of the maximum allowed motion acceleration max_m_s_s. The maximum allowed motion acceleration max_m_s_s can be related to the maximum second order rate of change of the delay [a, in m / s].

[0159] Fig.10f and 10g The two examples shown perform delay processing so that the delay compensates for variations in the listener-to-speaker distance between the speakers.

[0160] Auditory roughness can be alleviated by the following countermeasures:

[0161] Update VDL by interpolating the target delay value with sample accuracy (linearly interpolate from the current value to the target delay value at the end of each processing block)

[0162] The delay value returned by each output channel is used as the target value for the associated variable delay line, which applies the appropriate delay to the corresponding output signal. The implementation of these output delay lines is identical to the VDL used by distance rendering in MPEG-I.

[0163] Optionally, the gain is smoothed using single-pole averaging, see Fig.10h The calculation of the average constant is a function of the smoothing time constant gain_ms.

[0164] In a reverberant reproduction space, if the system or audio processor has been configured to optimize delay and / or gain without taking into account near and far fields, it is recommended that the system or audio processor can be configured to calibrate the gain and / or delay adjustments. The calibrated system option cal_system can be used when the system is running and the system has applied its own optimal gains and delays (etc.) for the sweet spot. In this case, see Figure 10i, we will additionally calculate the gain and delay compensation for the sweet spot (as above, see Fig.10c , which are calculated for the listener position). In this case, the difference between the two calculations is applied. In addition to these differences, the determination of the compensation gain shown in FIG10i is based on the Fig.10c The same considerations are described (like features are denoted by like reference numerals).

[0165] Audio Processing

[0166] For example, after calling rendering_gd_updatecfg, the rendering_gd_process function is called to specify the input and output buffers, see Fig.11a .

[0167] Optionally, the gain can be averaged using a single pole, see Fig.11b For example, the audio processor 10 described herein may be configured to perform a gain adjustment to determine the gain 41 based on the listener position. The gain adjustment may be performed by taking into account a target value (see gain0[ch]). The target value may represent a maximum allowable compensation gain, for example, which may be determined using the roll-off gain compensation function described herein, see Figure 4 , 6 , 10c and 10i. The current gain 41a, i.e. the gain determined for each loudspeaker without taking into account the different attenuation of the sound energy in the near field and the far field of each loudspeaker, is adjusted to a target value, i.e. a gain of 0 [ch], with a limited variation per time unit, i.e. per sample. When determining the target value, the different attenuation of the sound energy in the near field and the far field of each loudspeaker is taken into account. This prevents artifacts, since the gain changes only slightly per sample. The target value limits the gain variation and prevents too fast or erroneous gain variations due to irregular or too fast changes in the listener's position.

[0168] According to an embodiment, the delay of the external delay line can be calculated, see Fig.11c. The delay variation per frame and / or the second-order delay variation per frame is limited to reduce artifacts and pitch shifts. For example, the audio processor 10 described herein can be configured to perform delay processing to determine the delay 51 based on the listener position. The delay processing can be performed by considering a target value (see delay0[ch]). The target value can represent the delay of each speaker in the absence of boundary conditions, such as the delay of the current actual listener position, for example, without taking into account irregular or too fast changes in the listener position that may have occurred. The target value can be based on Fig.10d The delay determined in the delay processing of each speaker can be smoothed. For example, the audio processor can be configured to perform smoothing during the delay processing by smoothly transitioning from the delay of the previous frame (i.e., the frame before the current frame) determined for each speaker (see reference numeral 51a) to the delay of the current frame, for example, smoothly transitioning to a target value. Assuming that the speed and acceleration of the listener cannot exceed a certain value, a smooth delay is calculated, see reference numeral 51, see the consideration of delay_delta at the limit of the delay variation and / or the consideration of delay_delta2 at the limit of the second order of the delay variation. It may not be necessary to consider both limitations at the same time, but if both limitations are considered at the same time, artifacts can be more effectively reduced. The variable delay_delta represents the maximum number of samples allowed for the delay to vary from one frame to another, which can be calculated based on Fig.10f The variable delay_delta2 represents the maximum number of samples allowed for the delay to vary from one frame to another, which can be determined based on Figure 10g Thus, the maximum delay change rate and / or the maximum delay second-order change rate are limited in order to minimize artifacts.

[0169] The delay value returned by each output channel is used as the target value for the associated variable delay line, which applies the appropriate delay to the corresponding output signal. These output delay lines use the same implementation as the VDL.

[0170] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0171] The encoded audio signal of the present invention can be stored on a digital storage medium or transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0172] According to some implementation requirements, embodiments of the present invention can be implemented in hardware or software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, which has electronically readable control signals stored thereon, which cooperate with (or can cooperate with) a programmable computer system to perform the corresponding method.

[0173] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0174] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product is run on a computer, the program code is operable to perform one of the above methods.For example, the program code can be stored on a machine-readable carrier.

[0175] Other embodiments comprise the computer program for performing one of the methods herein, stored on a machine readable carrier.

[0176] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0177] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0178] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0179] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0180] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0181] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0182] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0183] The encoded audio signal of the present invention can be stored on a digital storage medium or transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0184] According to some implementation requirements, embodiments of the present invention can be implemented in hardware or software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, which has electronically readable control signals stored thereon, which cooperate with (or can cooperate with) a programmable computer system to perform the corresponding method.

[0185] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0186] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product is run on a computer, the program code is operable to perform one of the above methods.For example, the program code can be stored on a machine-readable carrier.

[0187] Other embodiments comprise the computer program for performing one of the methods herein, stored on a machine readable carrier.

[0188] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0189] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0190] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0191] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0192] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0193] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0194] The above embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the present invention be limited only by the scope of the patent claims to be presented, and not by the specific details presented herein by describing and explaining the embodiments.

[0195] References

[0196] [1] "Adaptively Adjusting the Stereophonic Sweet Spot to the Listener's Position",Sebastian Merchel and Stephan Groth,J.Audio Eng.Soc.,Vol.58,No.10,October 2010

[0197] [2] "AUDIO PROCESSOR, SYSTEM, METHOD AND COMPUTER PROGRAM FOR AUDIORENDERING", WO 2018 / 202324 A1

[0198] [3]https: / / www.princeton.edu / 3D3A / PureStereo / Pure_Stereo.html

[0199] [4]https: / / en.wikipedia.org / wiki / Critical_distance

Claims

1. An audio processor (10) for performing audio rendering by generating rendering parameters (100), the rendering parameters determining a loudspeaker signal (12) to be reproduced by a set of loudspeakers (14) from an audio signal (18), the audio processor being configured to: performing a gain adjustment to determine a gain (41) for generating a loudspeaker signal (12) for a loudspeaker (14) from an audio signal (18) based on a listener position (31), Obtain reverberation effect information (110); in, The audio processor (10) is configured to use a roll-off gain compensation function (42) for at least one speaker (14) in gain adjustment according to reverberation effect information (110), wherein the roll-off gain compensation function is used to map a listener-to-speaker distance (44) of the at least one speaker (14) to a listener-to-speaker distance compensation gain (46) of the at least one speaker (14), and for the roll-off gain compensation function, the compensated roll-off becomes monotonically shallower as the listener-to-speaker distance (44) increases.

2. The audio processor (10) of claim 1, wherein the reverberation effect information (110) indicates an amount of reverberation effective in a reproduction room (112) of the audio rendering, in, The roll-off gain compensation function (42) is adjusted so that as the listener-to-loudspeaker distance (44) increases, the compensated roll-off becomes more monotonically shallower and the amount of effective reverberation in the reproduction room (112) increases.

3. The audio processor (10) of claim 1 or 2, wherein the reverberation effect information (110) indicates whether the reverberation is effective in a reproduction room (112) of the audio rendering, in, The audio processor (10) is configured to use a roll-off gain compensation function (42) for which the compensated roll-off becomes monotonically shallower with increasing listener-to-loudspeaker distance (44) if the reverberation effect information (110) indicates that the reverberation is effective in the reproduction room (112) of the audio rendering, and to use a further roll-off gain compensation function for which the compensated roll-off is constant if the reverberation effect information (110) indicates that the reverberation is not effective in the reproduction room (112) of the audio rendering.

4. An audio processor (10) as described in any one of claims 1 to 3, wherein the roll-off gain compensation function (42) has a first compensation roll-off slope (421) within a first distance zone (441) and a second compensation roll-off slope (422) within a second distance zone (442), wherein the first compensation roll-off slope (421) is greater than the second compensation roll-off slope (422), and the first distance zone (441) involves a shorter distance than the second distance zone (422).

5. The audio processor (10) according to claim 4, configured to obtain a boundary distance (44) separating the first distance zone and the second distance zone from the reverberation effect information (110). 12 ).

6. The audio processor (10) according to claim 4 or 5, configured to derive the first compensation roll-off slope (421) and / or the second compensation roll-off slope (422) from the reverberation effect information (110).

7. An audio processor (10) as claimed in any one of claims 4 to 6, configured to obtain information from the reverberation effect information (110) about how the roll-off gain compensation function (42) transitions from the first distance zone to the second distance zone.

8. An audio processor (10) according to any one of the preceding claims, wherein the audio processor (10) is configured to perform gain adjustment so that the listener position (31) becomes an acoustically or perceptually optimal point relative to the set of loudspeakers (14).

9. An audio processor (10) according to any of the preceding claims, wherein the audio processor (10) is configured to perform delay processing so as to determine a delay (51) for generating a loudspeaker signal (12) for a loudspeaker (14) from an audio signal (18) based on a listener position (31).

10. The audio processor (10) of claim 9, wherein the audio processor (10) is configured to perform delay processing such that the delay (51) compensates for listener-to-speaker distance (44) variations between the speakers (14).

11. An audio processor (10) according to claim 9 or 10, wherein the audio processor (10) is configured to perform delay processing to make the listener position (31) an acoustically or perceptually optimal point relative to a set of loudspeakers (14).

12. The audio processor (10) according to any one of claims 9 to 11, wherein the audio processor (10) is configured to: performing delay processing by determining a delay (51) for each loudspeaker independently of a delay (51) determined for any other loudspeaker in a set of loudspeakers (14), or Delay processing is performed by determining a reference speaker in a set of speakers (14) and determining delays (51) of speakers (14) other than the reference speaker relative to the delay (51) determined for the reference speaker.

13. The audio processor (10) according to any one of claims 1 to 12, wherein a group of loudspeakers (14) belongs to one or more loudspeaker layers (15), and the audio processor (10) is configured to: If the sound source position (1041) of the desired audio signal is located between two speaker layers (15), For each of the two speaker layers (15), the speakers (14) in each speaker layer (15) applying a two-dimensional amplitude panning between the loudspeakers (14) and the loudspeakers (14) belonging to each loudspeaker layer (15) so as to determine a first panning gain (41) for the loudspeakers (14) belonging to each loudspeaker layer (15) for rendering of an audio signal (18) by the loudspeakers (14) belonging to each loudspeaker layer (15) from a virtual sound source position (104'1, 104"1) corresponding to a projection of a sound source position (1041) of a desired audio signal onto the respective loudspeaker layer (15), and Amplitude panning is applied between virtual sound source positions (104'1, 104"1) of two speaker layers (15) to determine a second panning gain (41) for the speaker layer (15) for rendering of an audio signal (18) by the speakers (14) of the two speaker layers from a desired sound source position (1041) of the audio signal when the second panning gain is applied in addition to the first panning gain (41).

14. The audio processor (10) according to any one of claims 1 to 13, wherein a group of loudspeakers (14) belongs to one or more loudspeaker layers (15), and the audio processor (10) is configured to: If the sound source position (1042) of the desired audio signal is located outside one or more speaker layers (15), A two-dimensional amplitude panning is applied between loudspeakers (14) belonging to a nearest loudspeaker layer (15) of the one or more loudspeaker layers (15) closest to a sound source position (1042) of a desired audio signal, so as to determine a first panning gain (41) for the loudspeakers (14) of the nearest loudspeaker layer (15) for obtaining a sound source position (1042) corresponding to the desired audio signal from the loudspeaker layer (15). rendering of an audio signal (18) by a loudspeaker (14) of the nearest loudspeaker layer (15) to a projected virtual sound source position (104'2) on the nearest loudspeaker layer (15), and applying a further amplitude panning together with a spectral shaping of the audio signal (18) between the loudspeakers (14) belonging to the nearest loudspeaker layer (15) in order to obtain a sound rendering by the loudspeakers (14) of the nearest loudspeaker layer (15) simulating a sound from a further virtual sound source position (104"2) deviating from the nearest loudspeaker layer (15) towards the desired sound source position (1042) of the audio signal, and A further amplitude panning is applied between the virtual sound source position (104′2) and the further virtual sound source position (104″2) in order to determine a second panning gain (41) for the panning between the virtual sound source position (104′2) and the further virtual sound source position (104″2) in order to obtain a rendering of the audio signal (18) by a loudspeaker (14) of the closest loudspeaker layer to the desired sound source position (1042) of the audio signal.

15. An audio processor (10) according to any one of claims 14, wherein the audio processor (10) is configured to, if the sound source position (1042) of the desired audio signal is located below one or more speaker layers (15), perform spectral shaping of the audio signal (18) using a first equalization function, the first equalization function simulating the timbre of bottom sound, and / or, if the sound source position (1042) of the desired audio signal is located above one or more speaker layers (15), perform spectral shaping of the audio signal (18) using a second equalization function, the second equalization function simulating the timbre of top sound.

16. The audio processor (10) according to any one of claims 1 to 15, wherein the audio processor (10) is configured to derive the reverberation effect information (110) from a bitstream.

17. The audio processor (10) according to any one of claims 1 to 16, wherein the audio processor (10) is configured to obtain the reverberation effect information (110) from side information of a bit stream, and to decode the audio signal (18) from the bit stream.

18. A method for audio rendering by generating rendering parameters (100) which determine a loudspeaker signal (12) to be reproduced by a set of loudspeakers (14) from an audio signal (18), the method comprising: performing a gain adjustment to determine a gain (41) for generating a loudspeaker signal (12) for a loudspeaker (14) from an audio signal (18) based on a listener position (31), Get reverberation effect information (110); In which, according to the reverberation effect information (110), the gain adjustment uses a roll-off gain compensation function (42) for at least one speaker, and the roll-off gain compensation function is used to map a listener-to-speaker distance (44) of the at least one speaker to a listener-to-speaker distance compensation gain (46) of the at least one speaker, and for the roll-off gain compensation function, the compensated roll-off becomes monotonically shallower as the listener-to-speaker distance (44) increases.

19. A computer program having a program code, for instructing the computer to execute the method according to claim 18 when the program is executed on the computer.

20. A bitstream (or a digital storage medium storing a bitstream) as claimed in any one of the preceding claims.

Images