Delay processing in audio rendering

Abstract

And an audio processor for performing audio rendering by generating rendering parameters that determine that speaker signals to be reproduced by the set of speakers are derived from the audio signal. The audio processor is configured to perform a delay process to determine a delay for generating a speaker signal for the speaker from the audio signal based on the listener position. Further, the audio processor is configured to control the delay processing by correcting a version of the listener location on which the delay processing begins or any intermediate value determined by the delay processing based on the listener location, thereby reducing artifacts in the audio presentation due to changes in the delay.

Description

[0001] manual Technical Field

[0002] Embodiments according to the invention relate to audio processors, systems, methods and computer programs for audio rendering, for example, user-adaptive speaker rendering using a tracking device. Background Art

[0003] A general problem with audio reproduction using loudspeakers is that the reproduction is usually only optimal for one or a small range of listener positions. Worse, the quality of the audio reproduction changes significantly when the listener changes position or moves. When the listening position changes away from the optimal point, the induced spatial auditory image is unstable. The stereo image collapses into the nearest loudspeaker.

[0004] This problem has been addressed in previous literature, including [1] by tracking the listener's position and adjusting gain and delay to compensate for deviations from the optimal listening position. [2] shows how to extend this to adapt to the spatial radiation characteristics of the speakers in use. Listener tracking is also used in conjunction with crosstalk cancellation (XTC), see for example [3]. XTC requires extremely precise positioning of the listener, so listener tracking is almost essential.

[0005] Previous methods for listener position-controlled delay adjustment / compensation of loudspeaker signals require smooth and precise tracking data to control a variable delay line (VDL) to adjust the loudspeaker signal delay. However, in real applications, listener movements can be highly dynamic and may include sudden changes in direction. Furthermore, the acquisition of position data may be affected by tracking errors, time jitter, and slow or irregular position update rates.

[0006] Therefore, it is desirable to devise a concept involving a delay adjustment scheme that is robust and capable of taking into account highly dynamic or imprecise tracking data inputs, minimizing perceived artifacts due to dynamic delay adjustments.

[0007] This object is achieved by the subject-matter of the independent claims.

[0008] Advantageous embodiments are the subject matter of the dependent claims. Summary of the invention

[0009] It has been found that controlling a variable delay line (VDL) by smooth and accurate listener tracking data is not always possible. However, a delay determined based on non-optimal listener position information may lead to artifacts in the audio reproduction. It is therefore an object of the present invention to provide a perceptually high quality delay adjustment / compensation which takes into account that real-world listener motion may be highly dynamic, may contain sudden changes of direction, and that the acquisition of position data may be affected by tracking errors, time jitter and slow position update rates. This difficulty may be overcome by using a modified version of the listener position or a modified intermediate value determined by the delay processing in the delay processing. The idea of ​​a basic embodiment of the present invention is that the input listener tracking data - or derived values ​​- may be modified, such as controlled, limited, smoothed or scaled, to avoid artifacts in the adaptive rendering. This is based on the recognition that the modification may reduce the variability / noise of the listener position information or the delay determined based on the listener information. Using the modified control of the delay processing, too fast and wrong changes of the delay are avoided or at least reduced, thereby reducing artifacts, even for very critical sound material, such as tones (high frequency sinusoidal tones, pitch pipes, glockenspiels). This enables listener-adaptive delay processing even in the real world where listener movement is highly dynamic and has sudden directional changes. Thus, when listening to sound reproduced by a set of speakers based on a signal or parameter derived from controlled delay processing, the listener can move within a large "sweet spot" (rather than a sweet spot) and experience a stable sound field within that large area.

[0010] Accordingly, embodiments relate to an audio processor that performs audio rendering by generating rendering parameters that determine speaker signals reproduced by a set of speakers from an audio signal. The audio processor is configured to perform delay processing to determine delays for generating speaker signals for the speakers from the audio signal based on a listener position. For example, each delay determined by the delay processing may be associated with one of the speakers, for example, the audio processor may be configured to determine a delay for each speaker based on which a delay for the respective speaker signal can be obtained. In addition, the audio processor is configured to control the delay processing by modifying a version of the listener position based on which the delay processing is started or any intermediate value determined by the delay processing based on the listener position, thereby reducing artifacts in the audio reproduction caused by changes in the delay.

[0011] The version of the listener position may be modified by adjusting, smoothing, clipping, or scaling the version of the listener position, such as the coordinates of the listener, the distance of the listener from one or more speakers in a set of speakers, the velocity of the listener, the acceleration of the listener, or the change in position between a previous listener position and a current listener position.

[0012] Any intermediate values ​​determined by the delay processing based on the listener position may be modified by adjusting, smoothing, clipping or scaling the intermediate values. The intermediate values ​​may be intermediate delay values. For example, for a speaker in a set of speakers, the distance between the listener and the speaker may be calculated and then converted to an intermediate delay value. Alternatively, the intermediate value may be a time rate of change of the delay, such as a time rate of change of the intermediate delay value, or a rate of change of a time rate of change of the delay, such as a rate of change of a time rate of change of the intermediate delay value.

[0013] The listener position may be defined by coordinates, e.g. tracking data, indicating the listener's position in the reproduction space, e.g. the position of the listener's body, the position of the listener's head, or the position of the listener's ear. For example, the listener position may be described using Cartesian coordinates, spherical coordinates, or cylindrical coordinates. In addition to the listener's absolute position, the listener position may also represent the listener's relative position, e.g. relative to a reference speaker in a set of speakers or relative to each speaker in a set of speakers or relative to an optimal point in the reproduction space or relative to any other predetermined position in the reproduction space.

[0014] The velocity of the listener, the acceleration of the listener, the time rate of change of the distance from the listener position to one or more speakers in a set of speakers, and the rate of change of the time rate of change of the distance from the listener position to one or more speakers in a set of speakers can represent different versions of the listener position.

[0015] The sweet spot may describe the focal point between loudspeakers where a listener perceives the sound reproduced by the loudspeakers as what a mixer intended them to be heard. The sweet spot may define the location within the reproduction space where all wavefronts emitted by a set of loudspeakers arrive at the same time. The sweet spot may also be referred to as a reference listening point.

[0016] According to an embodiment, the audio processor is configured to

[0017] For one or more of the following:

[0018] Listener position,

[0019] The speed of the listener,

[0020] the speed at which the listener moves toward one or more speakers in a set of speakers,

[0021] The listener's acceleration,

[0022] the acceleration of the listener towards one or more speakers in a set of speakers,

[0023] The distance from the listener position to one or more speakers in a set of speakers,

[0024] The time rate of change of the distance from the listener position to one or more speakers in a set of speakers,

[0025] The rate of change of the time rate of change of the distance from the listener position to one or more loudspeakers in a set of loudspeakers,

[0026] the delay of one or more speakers in a group of speakers,

[0027] the time rate of change of the delay of one or more speakers in a set of speakers, and

[0028] the rate of change of the time rate of change of the delay of one or more loudspeakers in a set of loudspeakers,

[0029] Perform one or more of the following to control the delay processing:

[0030] smooth

[0031] Cutting, and

[0032] Scaling is performed using a monotonically increasing function with a monotonically decreasing slope.

[0033] For example, the version of the listener position can be modified by smoothing, clipping and / or scaling the rate of change of the listener position, the listener's velocity, the listener's velocity toward one or more speakers in a set of speakers, the listener's acceleration, the listener's acceleration toward one or more speakers in a set of speakers, the distance from the listener position to one or more speakers in a set of speakers, the time rate of change of the distance from the listener position to one or more speakers in a set of speakers, and / or the time rate of change of the distance from the listener position to one or more speakers in a set of speakers. Intermediate values ​​determined by delay processing based on the listener position can be modified by smoothing, clipping and / or scaling delays (e.g., intermediate delay values), for example, intermediate delay values ​​for one or more speakers in a set of speakers, the time rate of change of the delays for one or more speakers in a set of speakers, and / or the rate of change of the time rate of change of the delays for one or more speakers in a set of speakers.

[0034] Such modifications can limit or control sudden and / or erroneous changes in listener position or delay. For example, smoothing can be applied to the listener position, where the listener position can be defined by tracking data. Thus, smoothing, for example, reduces tracking errors and time jitter, and smooth position data can be obtained even in the case of slow position update rates. For example, clipping constrains or limits values ​​to limit sudden changes. For example, the listener's velocity or the listener's acceleration can be clipped so that the listener's velocity or the listener's acceleration does not exceed a threshold. This also reduces dynamic delay adjustments determined based on the listener's position, thereby reducing artifacts that may appear due to too fast delay changes. The same applies to clipping the time rate of change of the distance of the listener position to one or more speakers of a set of speakers, or clipping the rate of change of the time rate of change of the distance of the listener position to one or more speakers of a set of speakers. The threshold value can correspond to a value at which the listener can perceive a pitch shift caused by too fast listener movement or a transient change in listener movement. In addition, the delay of one or more speakers in a group of speakers, the time rate of change of the delay of one or more speakers in a group of speakers, or the rate of change of the time rate of change of the delay of one or more speakers in a group of speakers can be directly clipped to limit / reduce the phase modulation caused by the delay. In addition, or alternatively, scaling can be applied to reduce or suppress values, especially values ​​related to rapid changes in the listener's position or rapid changes in the delay. Using a monotonically increasing function with a monotonically decreasing slope when scaling can make the scaling of large values ​​greater than that of small values. Therefore, the scaling of high changes or rapid changes is greater than the scaling of small changes or slow changes, wherein the monotonically decreasing slope makes the scaling of high speed, acceleration, delay, time rate of change of delay, or rate of change of time rate of change of delay greater than the scaling of small speed, acceleration, delay, time rate of change of delay, or rate of change of time rate of change of delay greater. This is conducive to reducing artifacts. Optionally, clipping and scaling can be used in combination, for example, the value can be scaled first, and if the scaled value still exceeds the threshold, the scaled value is clipped. Optionally, smoothing can be combined with clipping and scaling, for example, the listener position is first smoothed, and then the smoothed listener position or the value obtained therefrom is scaled and / or clipped. Alternatively, the value can also be scaled and / or clipped first, and then the scaled and / or clipped value can be smoothed, for example compared with a previous value, so that a smooth transition from the previous value to the current value can be obtained.

[0035] According to an embodiment, the audio processor is configured to control the delay processing according to the control information and to perform the correction according to the control information.

[0036] According to an embodiment, the audio processor is configured to derive from the control information one or more of information about the strength of the smoothing, information about a clipping threshold for clipping, and information about the parameterization of the monotonically increasing function with a monotonically decreasing slope. This optimizes the reduction of artifacts, since the delay processing can be individually controlled for different environments and loudspeakers, thereby optimizing the reduction of artifacts, since the control information can be provided individually for each environment or loudspeaker setup.

[0037] A further embodiment relates to a method for audio rendering by generating rendering parameters that determine speaker signals for reproduction by a set of speakers from an audio signal. The method includes performing delay processing to determine delays for generating the speaker signals for the speakers from the audio signal based on a listener position. Furthermore, the method includes controlling the delay processing by modifying a version of the listener position based on which the delay processing is started or any intermediate value determined by the delay processing based on the listener position, thereby reducing artifacts in the audio rendering due to delay variations.

[0038] A further embodiment relates to a digital storage medium storing a computer program or storing said computer program.The computer program has a program code, which instructs the computer to perform one of the methods described herein, when the program is executed on a computer.

[0039] Further embodiments relate to the bitstream described herein or a digital storage medium storing the bitstream.For example, the bitstream may include control information and / or audio signals and / or rendering parameters and / or delays and / or listener positions and / or loudspeaker signals and / or audio signals.

[0040] The methods, computer programs and bit streams described herein are based on the same considerations as the audio processors described herein. Incidentally, the methods, computer programs and bit streams can be implemented together with all the features and / or functions of the audio processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are not necessarily drawn to scale, with emphasis generally being placed on illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which:

[0042] Figure 1 A schematic diagram showing an embodiment of an audio processor that determines gain and delay;

[0043] Figure 2 A schematic diagram showing an embodiment of amplitude translation;

[0044] Figure 3 A schematic diagram showing an embodiment of an audio processor configured for delay adjustment;

[0045] Figure 4 A Level 1 processing system is shown as an example of an audio processor described herein;

[0046] Figure 5 shows an example of a roll-off gain compensation function,

[0047] Figure 6 The code snippet of the initialization phase is shown exemplarily;

[0048] Figure 7 The following code snippet shows the release phase:

[0049] Figure 8 The code snippet of the reset phase is exemplarily shown;

[0050] Figure 9a 9i to 9i show an exemplary code snippet of the real-time parameter update phase; and

[0051] Figures 10a to 10c The code snippet of the audio processing stage is shown as an example; DETAILED DESCRIPTION

[0052] The same or equivalent elements or elements having the same or equivalent functions are denoted by the same or equivalent reference numerals in the following description even if they appear in different drawings.

[0053] In the following description, a number of details are provided to provide a more comprehensive explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be implemented without these specific details. In other cases, in order to avoid obscuring embodiments of the present invention, well-known structures and devices are shown in block diagram form rather than in detail. In addition, unless otherwise specifically stated, the features of the different embodiments described below may be combined with each other.

[0054] Various examples are described below that help achieve more effective compression when using gain and / or delay adjustments controlled by listener position. For example, gain adjustment and / or delay adjustment can be added to other parameter adjustments for sound rendering, or can be provided separately.

[0055] In order to facilitate the understanding of the following examples of the present application, this description first introduces a possible device that can be matched with the examples of the present application summarized later. The following description first describes an embodiment of a device for generating speaker signals for multiple speakers. More specific embodiments and embodiments that can be applied individually or in groups are summarized below. Figure 1 Detailed description of the device.

[0056] Figure 1The apparatus is generally indicated using the reference numeral 10 and is used to generate speaker signals 12 for a plurality of speakers 14 in a manner that causes application of the speaker signals 12 across the plurality of speakers 14, or application of the speaker signals 12 to the plurality of speakers 14, to render at least one audio object at a desired virtual position.

[0057] The device 10 may be configured for a specific arrangement of speakers 14, i.e., a specific location where a plurality of speakers 14 are positioned or positioned and oriented. Alternatively, however, the device may be configurable for a different speaker arrangement of speakers 14. Likewise, the number of speakers 14 may be two or more, and the device may be designed for a predetermined number of speakers 14, or may be configurable for any number of speakers 14.

[0058] The device 10 includes an interface 16 at which the device 10 receives an audio signal 18 representing at least one audio object. For now, it is assumed that the audio input signal 18 is a monophonic audio signal representing an audio object such as the sound of a helicopter, etc. More examples and more details are provided below. Alternatively, the audio input signal 18 may be a stereophonic audio signal or a multi-channel audio signal. In any case, the audio signal 18 may represent an audio object in the time domain, the frequency domain, or any other domain, and it may represent the audio object in a compressed or uncompressed manner.

[0059] like Figure 1 As shown, the device 10 also includes an object position input 20 for receiving an expected virtual position 21. That is, at the object position input 20, the device 10 is informed about the expected virtual position 21, and the audio object will be virtually rendered to the virtual position by applying the speaker signal 12 at the speaker 14. That is, the device 10 receives information about the expected virtual position 21 at the input 20, and the information can be provided relative to the arrangement / position of the speaker 14, relative to the sweet spot, relative to the position and / or head direction of the listener and / or relative to the coordinates of the real world. The information can be based on a Cartesian coordinate system or a polar coordinate system. For example, it can be based on a room-centered coordinate system or a listener-centered coordinate system, which can be a Cartesian coordinate system or a polar coordinate system.

[0060] Alternatively, the apparatus 10 comprises a listener position input 30 for receiving an actual position of a listener. The listener position 31 may be defined by coordinates indicating the position of the listener within the reproduction space, such as the position of the listener's body, the position of the head or the position of the ear, such as tracking data, i.e. information of the listener's position over time. For example, the listener position 31 may be described using Cartesian coordinates, spherical coordinates or cylindrical coordinates. In addition to the absolute position of the listener, the listener position 31 may also represent a relative position of the listener, for example relative to a reference loudspeaker in a set of loudspeakers, or relative to an optimal point in the reproduction space, or relative to any other predetermined position in the reproduction space.

[0061] For example, in case the expected virtual position 21 defines the position of the audio object relative to the listener position 31 , the apparatus 10 may not necessarily need the listener position input 30 to receive the listener position 31 . This is because the virtual position 21 already takes the listener position 31 into account.

[0062] like Figure 1 As shown, the apparatus 10 may include a gain determiner 40 configured to determine gains 41 for the plurality of loudspeakers 14 based on the expected virtual position 21 received at the input 20 and / or the listener position 31 received at the input 30. According to an embodiment, the gain determiner 40 may calculate an amplitude gain for each loudspeaker signal so that the expected virtual position 21 is shifted between the plurality of loudspeakers 14 and / or so that roll-off of acoustic energy is compensated. As outlined in more detail below, see Figure 2 , the gain 41 may include a panning gain. The panning gain g applied to each loudspeaker signal n Can include horizontal component and the vertical component For example, The index n represents a positive integer in the range of 1≤n≤i, where i represents the number of speakers 14. The gain determiner 40 may be configured to determine a respective gain 41 for each speaker so as to render at least one audio object at a desired virtual position by applying the speaker signal 12 at or to the plurality of speakers 14.

[0063] Additionally or alternatively, the apparatus 10 may include a delay determiner / controller 50 to determine / control delays 51 for the plurality of speakers 14 based on the desired virtual position 21 received at the input 20 and / or the listener position 31 received at the input 30. The delay determiner 50 may be configured to determine a respective delay 51 for each speaker so that at least one audio object is rendered at a predetermined virtual position for or to a plurality of speakers 14, and / or so that the speaker signals reproduced by the speakers 14 arrive at the listener simultaneously.

[0064] The apparatus 10 may include an audio renderer 11 configured to render the audio signal 18 based on a gain 41 and / or a delay 51 , thereby deriving a speaker signal 12 from the audio signal 18 .

[0065] about Figure 2 , a possible 3D translation performed by the translation gain determiner 40 is described in more detail.

[0066] The loudspeakers 14 may be arranged in one or more horizontal layers 15. Figure 2 As shown, a first group of speakers 141 to 145 among the plurality of speakers 14 may be arranged in a first horizontal layer 151, and a second group of speakers 146 to 148 among the plurality of speakers 14 may be arranged in a second horizontal layer 152. That is, the first group of speakers 141 to 145 are similarly arranged at a similar height, and the second group of speakers 146 to 148 are similarly arranged at a similar height. The first group of speakers 141 to 145 may be arranged at a first height or close to a first height, and the second group of speakers 146 to 148 may be arranged at a second height or close to a second height, and the second height is, for example, higher than the first height. According to Figure 2 In the illustrated embodiment, the listener position 31 is exemplarily arranged in the first horizontal layer 151 .

[0067] In the following, an explanation will be given by taking a 3D rendered object as an example, in which an object 1041 (such as a sound source) is panned to a direction located between two physically existing speaker layers (at different heights) (seen from the listener 100). The object 1041 is panned in amplitude in the first layer 151 by sending an object signal to speakers in the layer with different first layer level gains (towards speakers 141 to 145), so that its amplitude is panned to the bottom layer, i.e., the first layer 151, see Figure 2 For example, in this horizontal panning, for each loudspeaker in the first group of loudspeakers 141 to 145, the horizontal component of the respective panning gain 41 is determined. Likewise, object 1041 is translated to Figure 2 For example, in this horizontal panning, for each speaker in the second group of speakers 146 to 148, the horizontal component of the respective panning gain 41 is determined. It can be seen that positions 104'1 and 104"1 can be selected so that they overlap each other vertically and / or so that the vertical projection of the expected position 1041 coincides with positions 104'1 and 104"1. Figure 2The rendering of the final object position 1041 by applying the amplitude translation between the layers 15 is shown, i.e., a vertical translation is shown. The virtual objects at the positions 104'1 and 104"1 are considered as virtual loudspeakers, and the amplitude translation by the gain determiner 40 is applied to render the virtual object at the desired position 1041 between the two layers 151 and 152. For example, in this vertical translation, a vertical component of the respective panning gain 41 is determined for each loudspeaker in the first group of loudspeakers 141 to 145 and the second group of loudspeakers 146 to 148. The result of the amplitude shift between layers 151 and 152 is two gain factors, namely the horizontal component of each speaker and the vertical component The loudspeaker signals are weighted by these two gain factors, for example, so that the sound source of the audio signal is panned to the desired sound source position of the audio signal. The weighting of the horizontal panning between the (real) loudspeaker layers 15 can also be frequency dependent, to compensate for the perceived effect that different frequency ranges in the vertical panning may have at different heights.

[0068] In the following, an example of a 3D rendered object is explained for the case where an object 1042 is translated above or below the outermost layer. The object may have a direction or position 1042 that is not within the direction range between the two layers 151 and 152 discussed with respect to the object position 1041. For example, the expected position 1042 of the object is above or below the (actually existing) layer 15, which here means below any available layer, in particular the lower layer, i.e., the first layer 151. For example, the object has been used as Figure 2 4 . In this case, a horizontal amplitude panning is applied by the panning gain determiner 40 to the bottom layer to render the object 1042 in the layer 151, see the resulting position 104'2. The resulting position 104'2 may represent a virtual sound source position corresponding to the projection of the sound source position (see 1042) of the desired audio signal onto the nearest speaker layer (see 151). More generally, the 2D amplitude panning is applied between the speakers 141 to 145 belonging to the speaker layer closest to the object 1042, i.e. the first layer 151. For example, when panning horizontally, for each speaker in the first group of speakers 141 to 145, the horizontal component of the respective panning gain 41 is determined. Further amplitude panning is then applied between the loudspeakers 141 to 145 belonging to the nearest loudspeaker layer, i.e. the first layer 151, while spectrally shaping the audio signal, thereby producing a sound presentation by the loudspeakers 141 to 145 of the nearest loudspeaker layer, i.e. the first layer 151, which simulates the sound from a further virtual sound source position 104"2 deviating from the nearest loudspeaker layer, i.e. the first layer 151, which is towards the desired sound source position of the audio signal, see 1042. Since there are no real loudspeakers in the vertical top or bottom directions, the vertical signal at 104"2 can be equalized to simulate the sound coloration of the top or bottom sound respectively. The vertical signal is then sent to the loudspeakers designated for the top / bottom directions. In order to render the final object position 1042, the panning gain determiner 40 may be configured to apply a further amplitude panning between the virtual sound source position 104'2 and the further virtual sound source position 104"2 to determine a second panning gain for panning between the virtual sound source position 104'2 and the further virtual sound source position 104"2, so that the speakers 141 to 145 of the nearest speaker layer render the audio signal from the sound source position 1042 of the desired audio signal. If the sound source position 1042 of the desired audio signal is located below one or more speaker layers, i.e. below the first layer 151, spectral shaping of the audio signal may be performed using a first equalization function that simulates the timbre of bottom sounds, and / or if the sound source position of the desired audio signal is located above one or more speaker layers, i.e. above the second layer 152, spectral shaping of the audio signal may be performed using a second equalization function that simulates the timbre of top sounds.

[0069] Figure 3 An embodiment of an audio processor 10 is shown for performing audio rendering by generating rendering parameters 100 , see audio renderer 11 , which determine deriving a loudspeaker signal 12 from an audio signal 18 to be reproduced by a set of loudspeakers 14 . Figure 3 The embodiment shown focuses on the delay determiner 50. Optionally, it can be combined with Figure 1 The gain determiner 40 is combined.

[0070] The audio processor 10 is configured to perform delay processing, see delay determiner 50, thereby determining a delay 51 based on a listener position 31 for generating a speaker signal 12 for a speaker 14 from an audio signal 18. The audio processor 10 is configured to control (see controller 52) the delay processing by modifying 52' a version 31 of the listener position based on which the delay processing is started, or by modifying 52" any intermediate value 54 determined by the delay processing based on the listener position 31, so as to reduce artifacts in the audio reproduction due to variations of the delay 51. The modification 52' or 52" can be performed by smoothing, clipping and / or scaling the corresponding input. Scaling can be performed using a monotonically increasing function with a monotonically decreasing slope.

[0071] The listener position version 31 may correspond to an absolute listener position within the reproduction space 112, a velocity of the listener, a velocity of the listener toward one or more speakers in a set of speakers 14, an acceleration of the listener, an acceleration of the listener toward one or more speakers in a set of speakers 14, a distance of the listener 1 to one or more speakers in a set of speakers 14, a time rate of change of the distance of the listener 1 to one or more speakers in a set of speakers 14, and / or a rate of change of a time rate of change of the distance of the listener 1 to one or more speakers in a set of speakers 14.

[0072] The intermediate value 54 may correspond to a delay of one or more speakers in a set of speakers 14 , a time rate of change of the delay of one or more speakers in a set of speakers 14 , and / or a rate of change of the time rate of change of the delay of one or more speakers in a set of speakers 14 .

[0073] The idea of ​​the basic embodiment of the invention is that limiting the variability (noise) of the listener position 31 (e.g., input listener tracking data) or the variability of derived values ​​(e.g., intermediate values ​​54) can be used to avoid artifacts in adaptive rendering, in particular the effects of a variable delay line (VDL). While avoiding artifacts associated with delay adjustments, it is still possible to react quickly enough to the movement of the listener. For example, the controller 52 can use movement speed and / or acceleration to control changes in the VDL operation.

[0074] Therefore, according to an embodiment, the above concept produces an audio processor, for example, comprising:

[0075] Means for controlling the delay processing of the loudspeaker audio signal 12, see controller 52, (aspect: "what to control / smooth?")

[0076] • Control criteria guide the above mentioned delay control / smoothing (aspect: “how to control / according to what criteria?”). For example, the controller 52 may obtain control parameters or control information, control corrections 52' and / or 52".

[0077] According to an embodiment, the audio processor 10 is configured to derive one or more of information about the intensity of smoothing, information about a clipping threshold for clipping and information about parameterization of a monotonically increasing function of a monotonically decreasing slope from the control information.

[0078] Devices for controlling delay processing: Inside the user adaptive renderer (see audio processor 10), according to an embodiment, the following calculations are performed for each speaker 14: User Location 3 1 (possibly jittery) calculates the distance between the user (i.e., listener 1) and each speaker 14 distance , for example calculated by the delay processing unit 55 of the delay determiner 50. The distance can then be converted into the corresponding Delay, for example, the median value is 54 (specified in samples at the sampling rate of the system or in seconds / milliseconds). This target delay can typically be used to control variable delay lines (VDLs) of the system. However, too fast or erroneous delay variations may lead to artifacts in the audio rendering. Therefore, since too fast a change of the listener position 31 or an erroneous listener position 31 will lead to too fast or erroneous delay variations, it is proposed to input such a delay into the controller 52 as an intermediate value 54 or to input a version of the listener position 31 directly into the controller 52. Therefore, there are several possibilities to limit the possible effects of too fast and erroneous variations.

[0079] The delay calculated in each processing frame can be smoothed / controlled, for example by modification 52" (most preferred variant);

[0080] o In a preferred variant thereof, the variation (difference) of the sampling delay from frame to frame is limited / controlled

[0081] Furthermore, for example, additionally or alternatively, the user-speaker distance calculated in each processing frame can be smoothed / controlled, for example by modifying 52″

[0082] o In a preferred variant thereof, the variation (difference) in user-speaker distance from frame to frame is limited / controlled. Also, for example, additionally or alternatively, the tracked user position used in each processed frame can be smoothed / controlled, for example by correction 52' (this is the least preferred variant)

[0083] o Specifically, the variation (difference) of the user position between frames can be limited / controlled

[0084] Control Standards : To control the delay processing (as described above), some intelligent criteria can be applied to ensure that the normal operation of the processor is not disturbed (i.e. the delay adjustment works in an optimal way to provide the same ideal sound quality to Listener 1 as in the sweet spot, as well asWhen the listener 1 moves in the room (i.e. the reproduction space 112) and the loudspeaker setup reacts fast enough. However, at the same time, even for very critical sound material, such as tonal sounds (high frequency sinusoidal tones, pitch pipes, glockenspiel), no artifacts should be generated due to time-varying delays.

[0085] As intelligent control criteria for delay processing, one or more parameters may be used, including:

[0086] Estimated listener velocity (in meters per second or other equivalent units). This can be measured by:

[0087] ο velocity in 3D space, or

[0088] o Speed ​​towards the speaker (possible, but less preferred)

[0089] Estimated listener acceleration (in meters per second 2 or other equivalent units)

[0090] ο acceleration in 3D space, or

[0091] o Acceleration towards the speaker (possible, but less preferred)

[0092] Alternatively, as a simple but less effective approach, the VDL action can be controlled by applying temporal smoothing directly to the variables in the calculation chain itself (see above), i.e.:

[0093] οVDL delay

[0094] οUser-speaker distance

[0095] ο Tracking user location

[0096] therefore, The audio signal processor 10 may include

[0097] Means for controlling the delay processing of the loudspeaker audio signal 18, such as a controller 52, (aspect: "what to control?")

[0098] ο See above (VDL latency, user-speaker distance, tracking user position)

[0099] · Control criteria that guide the above delay control (aspect: “how to control / according to what criteria?”)

[0100] ο see above (speed, acceleration)

[0101] οAlternative (see above): Direct smoothing of

[0102] VDL delay

[0103] User-speaker distance

[0104] Tracking user location

[0105] An embodiment according to the present invention relates to an audio processor 10, which is configured to generate a set of one or more parameters, namely rendering parameters 100, for each speaker in a set of one or more speakers 14 (for example, this can be a parameter that can affect the delay, level or frequency response of one or more audio signals 18). The rendering parameters determine the speaker signals 12 to be reproduced by each speaker 14 from the audio signal 18 based on the listener position 31 and the speaker position of the set of one or more speakers 14 (the listener position 31 can be the position of the entire body of the listener 1 in the same room (i.e. the reproduction space 112, such as the set of one or more speakers 14), or for example only the head position of the listener 1, or also for example the ear position of the listener 1. The listener position 31 can, for example, be a position relative to the set of one or more speakers 14, such as the distance of the listener's head to the set of one or more speakers 14.

[0106] Delay adjustment of the loudspeaker signal may be performed by a variable (fractional) delay line (VDL). While steady-state adjustment of the VDL is not critical, when interactively adjusting the VDL delay in dependence on user movement via a delay control signal, its dynamic behavior should be carefully limited to avoid perceptual impairment. Possible perceptual impairments arise from the fact that the dynamically adjusted delay line applies a phase modulation to the audio signal 18 that is processed in the delay line directed by the control signal.

[0107] Unconstrained phase modulation may result in audible roughness and / or perceptible pitch shifts of the pitch signal. The audible roughness results from rapid modulation within the control signal, e.g. caused by position tracking time jitter or sample-and-hold behavior of too slow or unstable position data acquisition. Perceptible pitch shifts or transient jumps in pitch shift may be caused by the user moving too fast or transient changes in the user's movement.

[0108] Therefore, one or more of the following countermeasures are perceptually beneficial:

[0109] Limitation of the allowed delay variation, e.g. by modifying 52", limiting the audible pitch deviation by limiting the instantaneous frequency modification

[0110] Limiting the allowed variation of delay variations, e.g. by modifying 52", to avoid audible pitch jumps by limiting instantaneous frequency jumps.

[0111] Adjust the delay in absolute rather than relative (relative to a dynamically selected reference channel with minimum delay), e.g. by correcting 52", to avoid unwanted instantaneous frequency jumps, especially due to listener movement around the sweet spot, where the reference channel would otherwise change abruptly.

[0112] Adjusting the absolute delay rather than the relative (relative to a dynamically selected reference channel with the smallest delay) delay, e.g. by correcting 52", minimizes the perceived pitch shift deviation of all channels combined, since the pitch shift can be up in one channel and down in another, thus staying centered around the true pitch, otherwise the reference channel would remain unchanged and only the other channels would be corrected in one direction.

[0113] Note that in addition to smoothing, clipping and / or scaling using a monotonically increasing function with a monotonically decreasing slope, interpolation may also be applied, for example by controller 52 at correction 52' or 52", so as to interpolate between frames. In other words, smoothing, clipping and / or scaling using a monotonically increasing function with a monotonically decreasing slope, like other tasks such as gain adaptation and translation, may be performed on a frame basis, and interpolation between consecutive smoothing / clipping / scaling values, i.e., values ​​of consecutive frames, may be used to change the delay in units finer than a frame, thereby going from a value in a previous frame to a value in a current frame.

[0114] An embodiment of the invention is now described for adaptive loudspeaker rendering.

[0115] First a general description should be made. As an alternative to rendering and binarizing MPEG-I scenes to headphones, playback through loudspeakers can be specified. In this mode of operation, the MPEG-I spatializer (HRTF-based renderer) is replaced by a dedicated loudspeaker-based renderer, as described below.

[0116] To achieve a high-quality listening experience, the loudspeaker setup assumes that the listener 1 is located at a dedicated fixed position, the so-called scan point. Typically, in a 6DOF playback situation, the listener 1 is moving. Therefore, the 3D spatial rendering must adapt immediately and continuously to the changing listener position 31. This can be achieved using two hierarchically nested layers of techniques.

[0117] 1. For example, gain 41 and delay 51 are applied to the loudspeaker signal 12 so that the signal 12 in the loudspeaker arrives at the listener position 31 with similar gain and delay, i.e., also at the optimal point. Optionally, a high-shelving compensation filter is applied to each loudspeaker signal 12 associated with the current listener position 31 and the direction of the loudspeaker relative to the listener 1. In this way, when the listener 1 moves to an off-axis position of the loudspeaker 14 or further, the high-frequency loss caused by the radiated high-frequency mode of the loudspeaker is compensated.

[0118] 2. Due to the 6DoF movement, the angles between the loudspeaker 14, the object and the listener 1 change as the listener position 31 changes. Therefore, the 3D amplitude translation algorithm (see Figure 2 ) will be updated in real time according to the relative position and angle of the changing listener position 31 and the fixed speaker configuration set in the LSDF. All coordinates (listener position 31, sound source position) can be transformed in the listening room coordinate system, that is, converted into the coordinate system of the reproduction space 112.

[0119] Physical compensation level (level 1)

[0120] Figure 4 An embodiment of a Level 1 system 10 is shown with an overview of its main components and parameters. Figures 1 to 3 The audio processor 10 may include Figure 4 The features and / or functions of the described embodiments.

[0121] Level 1 : Real-time update compensation of speaker (frequency-dependent) gain and delay, see Audio Renderer 11, to implement " Enhanced content rendering . By utilizing tracked user position information, such as a version of the listener position 31, the listener 1 (i.e., user) can move within a large "sweet spot" (rather than a sweet spot) and experience a stable sound field within that large area when listening to traditional content (e.g., stereo, 5.1, 7.1+4H). For immersive formats (i.e., non-stereo), when leaving the sweet spot, the sound appears to separate from the speakers 14 instead of collapsing into the nearest speaker 14, i.e., the quality is somewhat close to the effect of wave field synthesis, but limited to a single user experience. For stereo reproduction, the technology can provide stability of the left and right sound fields within a large range of the user position 31 (i.e., the range between left and right speakers at any distance).

[0122] For example, level 1 Gain Compensation It is based on the law of amplitude attenuation. In the free field, the amplitude is proportional to 1 / r, where r is the distance from the listener 1 to the loudspeaker 14 (1 / r corresponds to a 6 dB attenuation for each doubling of the distance). In the room 112, due to the presence of acoustic reflections and reverberation, the sound decays more slowly as the distance from the loudspeaker 14 increases. Therefore, near-field attenuation, far-field attenuation and / or critical distance parameters, such as those composed of the reverberation effect information 110, can be used to specify the relationship between the attenuation rate and the function of the distance from the loudspeaker 14. In addition, there may be a near-field-to-far-field transition parameter beta, such as those composed of the reverberation effect information 110. The larger beta is, the faster the transition between the near-field attenuation and the far-field attenuation. Figure 5An example of gain compensation as a distance function is shown, namely a roll-off gain compensation function 42, which may be used to determine a compensation gain to be applied by the audio renderer 11. In a reverberant field, the gain variation is smaller than in a free field.

[0123] For example, in level 1 Delay compensation The propagation delay from each speaker 14 to the listener position 31 is calculated and then a delay is applied to each speaker 14 to compensate for the propagation delay differences between the speakers 14. The delays may be normalized (with or without an offset) so that the minimum delay applied to the speaker signal 12 is zero.

[0124] Object Rendering Level (Level 2)

[0125] Level 2: User-tracked object translation can render point sources (objects, channels) in 6DoF playback space and requires Level 1 as a prerequisite. Therefore, it solves the use case of "6DoF VR / AR rendering". Level 1 system 10 can additionally include the following features and / or functions.

[0126] 3D amplitude panning algorithms can be used for speaker layers, such as horizontal and height layers, e.g. Figure 2 Each layer can apply a 2D translation algorithm to project objects onto that layer. The final 3D object is rendered by applying a magnitude translation between the two virtual objects of the 2D translation in the two layers.

[0127] When an object is above the top layer, a 2D pan is applied in that layer. The final 3D object is rendered by applying an amplitude pan between the 2D panned virtual object and the (non-existent) object in the vertical direction. The vertical object's signal can be equalized to mimic the timbre of the top sound and distributed evenly to the top layer's speakers.

[0128] When an object is below the lowest layer, a 2D pan is applied in that layer. The final 3D object is rendered by applying an amplitude pan between the virtual object and the (non-existent) object below it in the vertical direction. The signal of the vertical object can be equalized to simulate the timbre of the bottom sound and distributed evenly to the speakers of the lowest layer.

[0129] The vertical panning described is equally applicable to speaker setups with a single side (such as 5.1) and with multiple layers (such as 7.4.6).

[0130] Level 1 and Level 2 applied to object rendering render MPEG-I scenes faithfully as if they were rendered through headphones. This is a significant benefit compared to speaker-rendered MPEG-I content without applying adaptive tracking (1 and 2).

[0131] Physical compensation level (level 1)

[0132] An embodiment of gain and delay adjustment based on listener position is described below using code snippets, see Fig.9c To Figure 9i and Fig.10b and Fig.10c . Figure 1 Audio processor 10 or Figure 4 The Level 1 system 10 may include the features and / or functionality described below regarding gain and / or delay adjustment. Figure 3 The audio processor 10 may also include the features and / or functions regarding delay adjustment described below. Optionally, Figure 3 The audio processor 10 may include the features and / or functions regarding gain and / or delay adjustment described below. Optionally, Figure 1 Audio processor 10, Figure 3 The audio processor 10 and Figure 3 The audio processor 10 may include further features and / or functionalities as described below.

[0133] Data Elements and Variables

[0134] Below (see Figure 6-10c ) gives definitions and / or explanations of the data elements and variables used:

[0135] SFREQ_MIN minimum sampling rate [Hz] = 44100

[0136] SFREQ_MAX Maximum sampling rate [Hz] = 48000

[0137] VSOUND Speed ​​of sound in air [m / s] = 340.0

[0138] MAX_DELAY Maximum delay [samples] = 960

[0139] OVERHEAD_GAIN overhead [lin] = 0.25

[0140] framesize The number of samples per frame, default: 256

[0141] sfreq_Hz sampling frequency of input audio, default: 48000

[0142] nchan Number of channels (speakers)

[0143] max_delay maximum delay [samples], default: MAX_DELAY

[0144] bypass_on 0: normal operation, 1: bypass, default: 0

[0145] ref_proc 0: normal operation, 1: processing similar to the best point, default: 0

[0146] cal_system 0: normal operation, 1: calibration system, default: 0

[0147] gain_on 0: gain off, 1: gain on, default: 1

[0148] delay_on 0: delay off, 1: on, default: 1

[0149] decay_1_dB The near-field sound decay every time the distance doubles [dB], default: 8

[0150] decay_2_dB The far field sound decay every time the distance doubles [dB], default: 0

[0151] beta 1: default near-field-to-far-field conversion, >1 faster conversion

[0152] crit_dist_m critical distance [m], default: 4

[0153] max_m_s Maximum moving speed [v, unit m / s], default: 1

[0154] max_m_s_s Gain smoothing time constant [a, unit m / s], default: 1

[0155] gain_ms Gain smoothing time constant [ms], default: 40

[0156] sweet_spot sweet spot position [m,m,m]

[0157] spk_pos Speaker coordinates [m,m,m]

[0158] listener_pos listener coordinates [m,m,m]

[0159] For example, all coordinates are relative to the listening room defined in the LSDF file.

[0160] These parameters can be stored in the following structure:

[0161] Common data structures

[0162]

[0163] For example, the internal parameters calculated from the above parameters and states are stored in the following structure:

[0164] Internal data structures

[0165]

[0166]

[0167] Phase Description

[0168] The following describes an embodiment of gain and delay adjustment based on listener position using code snippets related to different stages. The embodiment may include an initialization stage (see Figure 6 ), release phase (see Figure 7 ), reset phase (see Figure 8 ), real-time parameter update phase (see Figure 9a to 9i) and audio processing stages (see Figures 10a to 10c ). Figure 1 Audio processor 10, Figure 4 Level 1 System 10 and Figure 3 The audio processor 10 may include the features and / or functions described for one or more stages, or individual features and / or functions of one or more stages.

[0169] initialization

[0170] Figure 6 The code snippet of the initialization phase is shown as an example.

[0171] Speaker setups can be loaded from LSDF files.

[0172] A structure of type rendering_gd_cfg_t is initialized with default values ​​and the nchan field is set to the number of speakers in the speaker setup.

[0173] The structure of type rendering_gd_rt_cfg_t is initialized with default values. The loudspeaker positions from the LSDF file are stored in the spk_pos field. If a ReferencePoint element is given in the LSDF file, its coordinates are stored in the sweet_spot field. If a calibration attribute is present, the field cal_system is set to the value of the calibration attribute.

[0174] The above structure is passed to the rendering_gd_init function.

[0175] release

[0176] Figure 7 The code snippet of the release phase is exemplarily shown.

[0177] Reset

[0178] Figure 8 The code snippet of the reset phase is exemplarily shown. Figure 8 Shows that all internal buffers have been flushed.

[0179] Update real-time parameters

[0180] In the update thread, the virtual listener position is converted to the listener room coordinate system. This is only relevant for VR scenes. In AR scenes, the two coordinate systems coincide.

[0181] All further processing happens in the audio thread.

[0182] Update the structure of type rendering_gd_rt_cfg_t by setting the listener_pos field to the listener position (in the listening room coordinate system), see Figure 9a This structure is then passed to the rendering_gd_updatecfg function, see Figure 9a .

[0183] Compensation gain and delay are calculated for each loudspeaker. Reference distance r_ref (at Figure 9a ) is the distance (dB, sample) at which the gain and delay compensation are zero. Based on the distance r from the speaker to the listener and the reference distance r_ref, the gain and delay compensation are calculated. The listener-to-speaker distance 44 is calculated based on the listener position 31 and the respective speaker position 32 as shown in Figure 9b The listener-to-loudspeaker distance 44 may represent a version of the listener position 31 .

[0184] In the free field, the sound decays by 6 dB for each doubling of the distance. In a room, the decay can be approximated by a smaller decay, for example 4 dB for each doubling of the distance. Alternatively, a critical distance (radius of the hall) can be considered. When approaching the loudspeaker, the decay is decay_dB for each doubling of the distance. Beyond the critical distance crit_dist_m, the sound decays only slowly. It is recommended to use the roll-off gain compensation function 42 (see Figure 5 and Fig.9c and FIG. 9i) to determine a gain compensation to compensate for the gain change caused by the sound attenuation.

[0185] Gain compensation can be based on the law of amplitude attenuation. In the free field, the amplitude is proportional to 1 / r, where r is the distance from the listener to the speaker (1 / r corresponds to a 6dB attenuation for every doubling of the distance). In the room, due to the presence of acoustic reflections and reverberation, the attenuation of sound slows down as the distance from the speaker increases. Therefore, near-field attenuation, far-field attenuation, and critical distance parameters can be used to specify the functional relationship between the attenuation rate and the speaker distance. In addition, there is a near-field-far-field conversion parameter beta47. The larger beta is, the faster the transition between near-field attenuation and far-field attenuation. The roll-off gain compensation function 42 may depend on the near-field-far-field conversion parameter beta 47. The near-field-far-field conversion parameter beta 47 can define how fast the roll-off gain compensation function 42 converts between the near field and the far field, that is, how fast the roll-off gain compensation function 42 converts from a steep increase in gain compensated by the listener-to-speaker distance 44 to a shallow increase or slight increase in gain compensated by the listener-to-speaker distance 44.

[0186] It is noted that the compensated roll-off becomes monotonically shallower as the listener-to-speaker distance 44 increases, which can be reflected by the slope of the compensated roll-off energy, which, when measured in the logarithmic domain, decreases monotonically with increasing listener-to-speaker distance 44.

[0187] The roll-off gain compensation function 42 maps a listener-to-speaker distance 44 associated with a loudspeaker to a listener-to-speaker distance compensation gain 41 of the loudspeaker associated with the listener-to-speaker distance 44. The roll-off gain compensation function 42 may be configured to compensate for a roll-off that becomes monotonically shallower as the listener-to-speaker distance 44 increases. As described above, in a reverberant reproduction space, the attenuation of sound energy in the near field and the far field may be different. Therefore, it is recommended to use a first attenuation parameter 481 (see decay_1_dB) for the near field (i.e., the first distance zone) and a second attenuation parameter 482 (see decay_2_dB) for the far field (i.e., the second distance zone), wherein the first distance zone is associated with a smaller listener-to-speaker distance 44 than the second distance zone. From Fig.9c As can be seen from FIG. 9i , the roll-off gain compensation function 42 takes into account the different attenuations 481 and 482 of the near field and the far field when determining the compensation gain 47 for a certain listener-to-speaker distance 44. For example, the roll-off gain compensation function 42 can take into account the degree of sound energy attenuation at the listener-to-speaker distance 44 according to a first attenuation parameter 481 (see pow_nf) and according to a second attenuation parameter 482 (see pow_ff). There is a critical distance 44 between the near field and the far field. 12 The acoustic energy attenuated according to the second attenuation parameter 482 (see pow_ff) may be scaled so that the acoustic energy attenuated according to the first attenuation parameter 481 and the second attenuation parameter 482 is at the critical distance 44.12 The first attenuation parameter 481 may represent a faster attenuation of the acoustic energy than the second attenuation parameter 482. Therefore, for the roll-off gain compensation function 42, the compensated roll-off becomes monotonically shallower as the listener-to-loudspeaker distance 44 increases.

[0188] Furthermore, the roll-off gain compensation function 42 may take into account how much sound energy is attenuated at the optimal point, see pow_ref at the optimal point r_ref. Thus, gain adjustment is performed so that the listener's position becomes the optimal point in an acoustic or perceptual sense relative to a set of loudspeakers. Considering the first attenuation parameter 481 and the second attenuation parameter 482, the sound energy attenuated at the optimal point may be determined.

[0189] Depending on the distance 44 between the loudspeaker and the listener position, the sound propagation time varies. These variations can be compensated by applying a delay. For example, an offset of MAX_DELAY / 2 is added to the compensation delay so that it is always positive, see Figure 9d Furthermore, the listener-to-loudspeaker distance may be taken into account when determining / adjusting the delays, as well as the distance between the sweet spot and the loudspeakers, see r_ref. Thus, the delay processing is performed so that the listener position is the sweet spot in an acoustic or perceptual sense relative to the set of loudspeakers.

[0190] Figure 9d It is shown that for each loudspeaker, a distance 44 from the listener position to the position of each loudspeaker can be determined, and based on the distance 44, a delay of each loudspeaker can be determined, see delay0[i].

[0191] from Figure 9d As can be seen, for each loudspeaker, an individual delay is determined, for example an absolute delay, see index i of delay variable delay0. Alternatively, the delay process may determine a reference loudspeaker in a set of loudspeakers and determine the delays of the other loudspeakers except the reference loudspeaker relative to the delay determined for the reference loudspeaker.

[0192] The overhead determined by OVERHEAD_GAIN can be used, see Fig.9e That is, when the listener is far away from the loudspeaker, the system can amplify the signal by up to 1 / OVERHEAD_GAIN. If the gain exceeds this value, the gain of all channels is scaled by the same factor so that the maximum gain is 1.0 (0dB). This is equivalent to the action of the inter-channel ganged limiter.

[0193] In addition to gain adjustments, delay adjustments can also be performed to reduce artifacts in the audio presentation due to delay variations.

[0194] According to an embodiment, the control of the delay processing can be performed by tailoring the listener's speed or by tailoring the delay, wherein the tailoring of the delay and the listener's speed can be controlled based on the maximum allowed listener speed (see max_m_s). For example, a maximum speed can be defined at which delay changes due to too fast a change in the listener's position will hardly produce artifacts in the audio rendering. Figure 9f The maximum delay variation (see delay_delta) determined based on the maximum allowed listener velocity is shown. The number of samples that the delay is allowed to vary from one frame to another is calculated as a function of the maximum allowed movement velocity max_m_s. The maximum allowed movement velocity max_m_s can be related to the maximum delay variation rate [v, in m / s].

[0195] According to alternative embodiments, the control of the delay processing may be performed by clipping the listener's acceleration or by clipping the time rate of change of the delay, wherein the time rate of change of the delay and the clipping of the listener's acceleration may be controlled based on a maximum allowed listener acceleration (see max_m_s_s). For example, a maximum acceleration may be defined for which delay changes due to too fast a change in the listener's position will hardly produce artifacts in the audio rendering. Figure 9g The maximum time rate of change of the delay determined based on the maximum allowed listener acceleration is shown, see delay_delta2. The number of samples by which the delay is allowed to change from one frame to another is calculated as a function of the maximum allowed motion acceleration max_m_s_s. The maximum allowed motion acceleration max_m_s_s can be related to the maximum second order rate of change of the delay [a, in m / s].

[0196] Figure 9f and 9g The two examples shown perform delay processing so that the delay compensates for variations in the listener-to-speaker distance between the speakers.

[0197] Auditory roughness can be alleviated by the following countermeasures:

[0198] Update VDL by interpolating the target delay value with sample accuracy (linearly interpolate from the current value to the target delay value at the end of each processing block)

[0199] The delay value returned by each output channel is used as the target value for the associated variable delay line, which applies the appropriate delay to the corresponding output signal. The implementation of these output delay lines is identical to the VDL used by distance rendering in MPEG-I.

[0200] Optionally, the gain is smoothed using single-pole averaging, see Figure 9h The calculation of the average constant is a function of the smoothing time constant gain_ms.

[0201] In a reverberant reproduction space, if the system or audio processor has been configured to optimize delay and / or gain without taking into account near and far fields, it is recommended that the system or audio processor can be configured to calibrate the gain and / or delay adjustments. The calibrate system option cal_system can be used when the system is running and the system has applied its own optimal gain and delay (etc.) for the sweet spot. In this case, see Figure 9i, we will additionally calculate the gain and delay compensation for the sweet spot (Figure 9i above). Fig.9c , which are calculated for the listener position). In this case, the difference between the two calculations is applied. In addition to these differences, the determination of the compensation gain shown in FIG. 9i is based on the Fig.9c The same considerations are described (like features are denoted by like reference numerals).

[0202] Audio Processing

[0203] For example, after calling rendering_gd_updatecfg, the rendering_gd_process function is called to specify the input and output buffers, see Fig.10a .

[0204] Optionally, the gain can be averaged using a single pole, see Fig.10b For example, the audio processor 10 described herein may be configured to perform a gain adjustment to determine the gain 41 based on the listener position. The gain adjustment may be performed by taking into account a target value (see gain0[ch]). The target value may represent a maximum allowable compensation gain, for example, which may be determined using the roll-off gain compensation function described herein, see Figure 5 , 9c and 9i. The current gain 41a, i.e. the gain determined for each loudspeaker without taking into account the different attenuation of the sound energy in the near field and in the far field of each loudspeaker, is adjusted to a target value, i.e. a gain of 0 [ch], with a limited variation per time unit, i.e. per sample. When determining the target value, the different attenuation of the sound energy in the near field and in the far field of each loudspeaker is taken into account. This prevents artifacts, since the gain changes only slightly per sample. The target value limits the gain variation and prevents too rapid or erroneous gain variations due to irregular or too rapid changes in the listener's position.

[0205] According to an embodiment, the delay of the external delay line can be calculated, see Fig.10c. The delay variation per frame and / or the second-order delay variation per frame is limited to reduce artifacts and pitch shifts. For example, the audio processor 10 described herein can be configured to perform delay processing to determine the delay 51 based on the listener position. The delay processing can be performed by considering a target value (see delay0[ch]). The target value can represent the delay of each speaker in the absence of boundary conditions, such as the delay of the current actual listener position, for example, without taking into account irregular or too fast changes in the listener position that may have occurred. The target value can be based on Figure 9d The delay determined in the delay processing of each speaker can be smoothed. For example, the audio processor can be configured to perform smoothing during the delay processing by smoothly transitioning from the delay of the previous frame (i.e., the frame before the current frame) determined for each speaker (see reference numeral 51a) to the delay of the current frame, for example, smoothly transitioning to a target value. Assuming that the speed and acceleration of the listener cannot exceed a certain value, a smooth delay is calculated, see reference numeral 51, see the consideration of delay_delta at the limit of the delay variation and / or the consideration of delay_delta2 at the limit of the second order of the delay variation. It may not be necessary to consider both limitations at the same time, but if both limitations are considered at the same time, artifacts can be more effectively reduced. The variable delay_delta represents the maximum number of samples allowed for the delay to vary from one frame to another, which can be calculated based on Figure 9f The variable delay_delta2 represents the maximum number of samples allowed for the delay to vary from one frame to another, which can be determined based on Figure 9g Thus, the maximum delay change rate and / or the maximum delay second-order change rate are limited in order to minimize artifacts.

[0206] The delay value returned by each output channel is used as the target value for the associated variable delay line, which applies the appropriate delay to the corresponding output signal. These output delay lines use the same implementation as the VDL.

[0207] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0208] The encoded audio signal of the present invention can be stored on a digital storage medium or transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0209] According to some implementation requirements, embodiments of the present invention can be implemented in hardware or software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, which has electronically readable control signals stored thereon, which cooperate with (or can cooperate with) a programmable computer system to perform the corresponding method.

[0210] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0211] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product is run on a computer, the program code is operable to perform one of the above methods.For example, the program code can be stored on a machine-readable carrier.

[0212] Other embodiments comprise the computer program for performing one of the methods herein, stored on a machine readable carrier.

[0213] In other words, therefore, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0214] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0215] Therefore, a further embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0216] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0217] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0218] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0219] The above embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the present invention be limited only by the scope of the patent claims to be presented, and not by the specific details presented herein by describing and explaining the embodiments.

[0220] References

[0221] [1] "Adaptively Adjusting the Stereophonic Sweet Spot to the Listener's Position",Sebastian Merchel and Stephan Groth,J.Audio Eng.Soc.,Vol.58,No.10,October 2010

[0222] [2] "AUDIO PROCESSOR, SYSTEM, METHOD AND COMPUTER PROGRAM FOR AUDIORENDERING", WO 2018 / 202324 A1

[0223] [3]https: / / www.princeton.edu / 3D3A / PureStereo / Pure_Stereo.html

Claims

1. An audio processor (10) for performing audio rendering by generating rendering parameters (100), the rendering parameters determining a loudspeaker signal (12) to be reproduced by a set of loudspeakers (14) from an audio signal (18), the audio processor being configured to: performing delay processing to determine a delay (51) for generating a loudspeaker signal (12) for a loudspeaker (14) from an audio signal (18) based on a listener position (31), in, The audio processor (10) is configured to control the delay processing by modifying (52', 52") a version of the listener position (31) based on which the delay processing is initiated or any intermediate value (54) determined based on the listener position (31) so as to reduce artifacts in the audio presentation caused by variations in the delay (51).

2. The audio processor (10) according to claim 1, wherein the audio processor (10) is configured to perform control of the delay processing by: For one or more of the following: Listener position (31), The speed of the listener, the speed at which the listener moves towards one or more speakers in a set of speakers (14), The listener's acceleration, the acceleration of the listener towards one or more speakers in a set of speakers (14), the distance from the listener position (31) to one or more speakers in a set of speakers (14), a time rate of change of the distance from the listener position (31) to one or more speakers in a set of speakers (14), a rate of change of the time rate of change of the distance from the listener position (31) to one or more speakers in a set of speakers (14), a delay of one or more loudspeakers in a set of loudspeakers (14), a time rate of change of delay of one or more loudspeakers in a set of loudspeakers (14), and a rate of change of the time rate of change of the delay of one or more loudspeakers in a set of loudspeakers (14), Perform one or more of the following: smooth Cutting, and Scaling is performed using a monotonically increasing function with a monotonically decreasing slope.

3. An audio processor (10) according to claim 1 or 2, wherein the audio processor (10) is configured to perform delay processing so that the delay (51) compensates for changes in the listener-to-speaker distance (44) between the speakers (14).

4. An audio processor (10) according to any one of claims 1 to 3, wherein the audio processor (10) is configured to perform delay processing so that the listener position (31) becomes the best point in an acoustic or perceptual sense relative to a set of speakers (14).

5. An audio processor (10) according to any one of claims 1 to 4, wherein the audio processor (10) is configured to perform gain adjustment so as to determine a gain (41) for generating a loudspeaker signal (12) of a loudspeaker (14) from an audio signal (18) based on a listener position (31).

6. An audio processor (10) according to any one of claims 1 to 5, wherein the audio processor (10) is configured to perform gain adjustment by using a roll-off gain compensation function (42) for each speaker, the roll-off gain compensation function being used to map a listener-to-speaker distance (44) of each speaker to a listener-to-speaker distance compensation gain (46) of each speaker.

7. The audio processor (10) of claim 6, wherein the audio processor (10) is configured to perform gain adjustment so that the listener position (31) becomes an acoustically or perceptually optimal point relative to the set of loudspeakers (14).

8. The audio processor (10) according to any one of claims 1 to 7, wherein a group of loudspeakers (14) belongs to one or more loudspeaker layers (15), and the audio processor (10) is configured to: If the sound source position (1041) of the desired audio signal is located between two speaker layers (15), for each of the two speaker layers (15), applying a two-dimensional amplitude panning between the speakers (14) of the respective speaker layer (15) so as to determine a first panning gain (41) for the speakers (14) belonging to the respective speaker layer (15) for rendering of an audio signal (18) by the speakers (14) belonging to the respective speaker layer (15) from a virtual sound source position (104'1, 104"1) corresponding to a projection of a sound source position (1041) of the desired audio signal onto the respective speaker layer (15), and Amplitude panning is applied between virtual sound source positions (104'1, 104"1) of two speaker layers (15) to determine a second panning gain (41) for the speaker layer (15) for rendering of an audio signal (18) by the speakers (14) of the two speaker layers from a desired sound source position (1041) of the audio signal when the second panning gain is applied in addition to the first panning gain (41).

9. The audio processor (10) according to any one of claims 1 to 8, wherein a group of loudspeakers (14) belongs to one or more loudspeaker layers (15), and the audio processor (10) is configured to: If the sound source position (1042) of the desired audio signal is located outside one or more speaker layers (15), applying a two-dimensional amplitude panning between the loudspeakers (14) belonging to a nearest loudspeaker layer (15) closest to a sound source position (1042) of a desired audio signal in one or more loudspeaker layers (15) so as to determine a first panning gain (41) for the loudspeakers (14) of the nearest loudspeaker layer (15) for rendering of an audio signal (18) by the loudspeakers (14) of the nearest loudspeaker layer (15) from the sound source position (1042) corresponding to the desired audio signal to a projected virtual sound source position (104'2) on the nearest loudspeaker layer (15), and applying a further amplitude panning together with a spectral shaping of the audio signal (18) between the loudspeakers (14) belonging to the nearest loudspeaker layer (15) in order to obtain a sound rendering by the loudspeakers (14) of the nearest loudspeaker layer (15) simulating a sound from a further virtual sound source position (104"2) deviating from the nearest loudspeaker layer (15) towards the desired sound source position (1042) of the audio signal, and A further amplitude panning is applied between the virtual sound source position (104′2) and the further virtual sound source position (104″2) in order to determine a second panning gain (41) for the panning between the virtual sound source position (104′2) and the further virtual sound source position (104″2) in order to obtain a rendering of the audio signal (18) by a loudspeaker (14) of the closest loudspeaker layer to the desired sound source position (1042) of the audio signal.

10. An audio processor (10) according to any one of claims 9, wherein the audio processor (10) is configured to, if the sound source position (1042) of the desired audio signal is located below one or more speaker layers (15), perform spectral shaping of the audio signal (18) using a first equalization function, the first equalization function simulating the timbre of bottom sound, and / or, if the sound source position (1042) of the desired audio signal is located above one or more speaker layers (15), perform spectral shaping of the audio signal (18) using a second equalization function, the second equalization function simulating the timbre of top sound.

11. The audio processor (10) according to any one of the preceding claims, wherein the audio processor (10) is configured to: performing delay processing by determining a delay for each loudspeaker independently of a delay determined for any other loudspeaker in a set of loudspeakers (14), or Delay processing is performed by determining a reference speaker in a set of speakers (14) and determining delays (51) of speakers (14) other than the reference speaker relative to the delay determined for the reference speaker.

12. The audio processor (10) according to any one of the preceding claims, wherein the audio processor (10) is configured to: The delay processing is performed by determining a delay for each speaker independently of a delay determined for any other speaker in a set of speakers (14) so ​​as to obtain an absolute delay for each speaker, wherein the audio processor (10) is configured to For one or more of the following: the absolute delay of one or more loudspeakers in a set of loudspeakers (14), the time rate of change of the absolute delay of one or more loudspeakers in a set of loudspeakers (14), and The rate of change of the time rate of change of the absolute delay of one or more speakers in a set of speakers (14) is controlled by performing one or more of the following processing: smooth Cutting, and Scaling is performed using a monotonically increasing function with a monotonically decreasing slope.

13. The audio processor (10) according to any one of the preceding claims, wherein the audio processor (10) is configured to: Delay processing is performed by determining the distance from the listener position (31) to the position of each speaker for each speaker, and determining the delay of each speaker based on the distance.

14. The audio processor (10) of claim 13, wherein the audio processor (10) is configured to: the distance from the listener position (31) to one or more speakers in a set of speakers (14), a time rate of change of the distance from the listener position (31) to one or more speakers in a set of speakers (14), a rate of change of the time rate of change of the distance from the listener position (31) to one or more speakers in a set of speakers (14), Perform one or more of the following to control the delay processing: smooth Cutting, and Scaling is performed using a monotonically increasing function with a monotonically decreasing slope.

15. The audio processor (10) according to any one of the preceding claims, wherein the audio processor (10) is configured to control the delay processing in dependence on control information and to perform the modification in dependence on the control information.

16. The audio processor (10) according to any one of claims 2, 12 and 14, wherein the audio processor (10) is configured to obtain one or more of the following from the control information: Information about the strength of the smoothing, Information about the clipping threshold for clipping, Information about the parameterization of a monotonically increasing function with a monotonically decreasing slope.

17. The audio processor (10) of claim 15 or 16, wherein the audio processor (10) is configured to derive the control information from the bitstream.

18. The audio processor (10) according to claim 15 or 16, wherein the audio processor (10) is configured to derive the control information from side information of a bit stream and to decode the audio signal (18) from the bit stream.

19. A method for audio rendering by generating rendering parameters (100) which determine a loudspeaker signal (12) to be reproduced by a set of loudspeakers (14) from an audio signal (18), the method comprising: performing delay processing to determine a delay (51) for generating a loudspeaker signal (12) for a loudspeaker (14) from an audio signal (18) based on a listener position (31), The delay processing is controlled by modifying a version of the listener position (31) based on which the delay processing is initiated or any intermediate value (54) determined based on the listener position (31) so as to reduce artifacts in the audio presentation caused by variations in the delay (51).

20. A computer program having a program code (or a digital storage medium storing the computer program), which, when the program is executed on a computer, instructs the computer to execute the method according to claim 19.

21. A bitstream (or a digital storage medium storing said bitstream) as claimed in any preceding claim.

Images