Apparatus and method for automatic control of reverberation levels using a perceptual model - Patent Application 20070123633

Abstract

According to one embodiment, an apparatus is provided for processing an audio input signal comprising one or more audio channels to obtain an audio output signal. The apparatus comprises a reverberation gain determiner (110) configured to determine reverberation gain information in response to the audio input signal. Furthermore, the apparatus comprises a signal processor (120) configured to obtain the audio output signal in response to the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

Description

[Technical field]

[0001] The present invention relates to audio signal processing, and in particular to an apparatus and method for automatic control of reverberation levels, for example by using a perceptual model. [Background technology]

[0002] Reverberation is a very complex element in acoustics and audio signal processing, and its perceived intensity and its control are particularly important. Audio signals, such as music recordings or radio broadcasts, can be processed, for example, by artificial reverberation to emulate the acoustic characteristics of a particular environment, for example a concert venue or hall. The input signal can be, for example, a mixture of direct and ambient signals. Direct signals can be, for example, recordings of singing, musical instruments, and sound events, such as gunshots and alarm sirens. The term "direct sounds" indicates that these sounds are directional and can be localized as coming from one direction. Ambient (or diffuse) sound components can be, for example, reverberation and environmental sounds that are not perceived as coming from a specific direction, such as wind noise and rain. In music recordings, reverberation is the most prominent source of ambient sound.

[0003] The perceived reverberation level depends on the input signal and the reverberation impulse response, e.g., the pre-delay and the length of the reverberation tail (see [1]). Non-stationary input signals with transients and fast attacks of the subband envelope, e.g., drum sounds, will produce higher reverberation intensity.

[0004] An input signal with a rapidly decaying envelope and silences is less effective at masking reverberation (see [2]). With small reverberation times and pre-delays, the input signal will match the reverberant signal and partially mask it to a greater extent, as with large reverberation times and pre-delays. Furthermore, the acoustic environment (see [3]), the playback system (see [4]), and other aesthetic aspects related to the musical genre influence the preferred gain parameter settings.

[0005] A model for predicting several spatial attributes conditioned on listening test data is described in [5], but the proposed model was not applied in real time and the model itself was also more complex, since it used up to 12 audio features, while resulting in a performance comparable to the proposed algorithm using only three features. The model of the perceived level of late reverberation in (see [6]) uses the partial loudness of the direct and reverberant components of the input audio signal. A study on the perceived level of artificial late reverberation in [1] showed that the perceived level depends on the reverberation time and the input signal, but not on the inter-channel correlation of the impulse response, and that the evaluation differences between individuals and repeated tests of the same individual were similar. The preferred level is investigated in [7], where it is concluded that the aesthetic quality is worse when the applied reverberation is above the preferred level than below it. Summary of the Invention [Problem to be solved by the invention]

[0006] The object of the present invention is to provide an improved concept for automatically controlling reverberation levels. This object is solved by an apparatus according to claim 1, a method according to claim 19 and a computer program according to claim 20. [Means for solving the problem]

[0007] According to an embodiment, an apparatus is provided for processing an audio input signal comprising one or more audio channels to obtain an audio output signal, the apparatus comprising a reverberation gain determiner configured to determine reverberation gain information in response to the audio input signal, and a signal processor configured to obtain the audio output signal in response to the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

[0008] Further, according to an embodiment, a method for processing an audio input signal to obtain an audio output signal is provided, the method comprising the steps of: determining reverberation gain information in response to an audio input signal; - obtaining an audio output signal depending on reverberation gain information by adding artificial reverberation to the audio input signal or to a preprocessed audio signal that is dependent on the audio input signal; Contains:

[0009] Furthermore, a computer program is provided for carrying out the above-described method when executed on a computer or signal processor.

[0010] In the following, embodiments of the invention are explained in more detail with reference to the drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an apparatus for processing an audio input signal comprising one or more audio channels to obtain an audio output signal according to one embodiment. [Diagram 2] FIG. 2 is a block diagram of a reverberation control algorithm according to one embodiment. [Diagram 3] FIG. 1 illustrates the average rating per listening test item and the grand mean with 95% confidence intervals according to one embodiment. [Figure 4] FIG. 2 illustrates predicted reverberation strength versus observed reverberation strength according to one embodiment. [Diagram 5] The top part of FIG. 5 shows a spectrogram of an audio signal that transitions from classical music to pop music after 25 seconds, the middle part of FIG. 5 shows the equivalent reverberation level before and after temporal smoothing, and the bottom part of FIG. 5 shows the smoothing factor and total scaling value according to one embodiment. [Figure 6] FIG. 13 illustrates reference reverberation transmit gain across equivalent reverberation levels with fitted polynomials according to one embodiment. [Figure 7]FIG. 1 illustrates an equivalent reverberation level versus reverberation transmit gain error according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 illustrates an apparatus for processing an audio input signal comprising one or more audio channels to obtain an audio output signal according to one embodiment.

[0013] The apparatus comprises a reverberation gain determiner 110 configured to determine reverberation gain information in response to an audio input signal.

[0014] The apparatus further comprises a signal processor 120 configured to obtain an audio output signal depending on the reverberation gain information by adding artificial reverberation to the audio input signal or to a preprocessed audio signal that is dependent on the audio input signal.

[0015] According to one embodiment, the reverberation gain determiner 110 may be configured to determine the reverberation gain information in response to, for example, an estimate of the perceived intensity of reverberation in the audio input signal.

[0016] In one embodiment, the reverberation gain determiner 110 may be configured to determine the reverberation gain information, for example, by employing a model that, upon receiving information regarding one or more characteristics of the audio input signal, returns an estimate of the perceived intensity of reverberation in the audio input signal.

[0017] According to one embodiment, the model employed by the reverberation gain determiner 110 may for example be a linear regression model using one or more feature values ​​of one or more features of the audio input signal as input for the linear regression model.

[0018] In one embodiment, to obtain the reverberation transmit gain, the reverberation gain determiner 110 may be configured to determine the reverberation transmit gain as reverberation gain information, for example by mapping an estimate of the perceived intensity of reverberation in the audio input signal according to a mapping function. An equivalent reverberation intensity (=the intensity perceived after application of a scaling value) may for example be fed to the mapping function. For example, an estimate of the perceived intensity of reverberation may be mapped to listening test data and the respectively fitted curve may for example then be used for conversion to the reverberation transmit gain.

[0019] According to one embodiment, the one or more features of the audio input signal may, for example, depend on inter-channel correlation of at least one of one or more subbands of two audio channels of the one or more audio channels of the audio input signal.

[0020] In one embodiment, the one or more characteristics of the audio input signal may, for example, depend on a spectral flatness measure of at least one of one or more subbands of one or more audio channels of the audio input signal.

[0021] According to an embodiment, the reverberation gain determiner may be configured to determine an estimate of a perceived intensity of reverberation in the audio input signal, for example by using a model. The reverberation gain determiner may be configured to determine one or more scaling factors depending on one or more features of the audio input signal. Furthermore, the reverberation gain determiner may be configured to determine reverberation gain information depending on the estimate of the perceived intensity of reverberation and depending on the one or more scaling factors.

[0022] In one embodiment, the one or more scaling factors may depend, for example, on inter-channel correlation of at least one of one or more subbands of two audio channels of one or more audio channels of the audio input signal.

[0023] According to one embodiment, the one or more scaling factors may, for example, depend on the presence of transient signal components in at least one of the one or more audio channels of the audio input signal.

[0024] In one embodiment, the one or more scaling factors may depend, for example, on a spectral transient measure in at least one of the one or more audio channels of the audio input signal. TIFF2025509606000002.tif14132, where: TIFF2025509606000003.tif2182, TIFF2025509606000004.tif34 indicates the time index, TIFF2025509606000005.tif54 indicates the frequency index, TIFF2025509606000006.tif68 is the frequency band / subband TIFF2025509606000007.tif43, TIFF2025509606000008.tif827 represents a frequency band of at least one of one or more audio channels or a combination of at least one of one or more audio channels. One or more frequency bins in TIFF2025509606000009.tif43 Time index for TIFF2025509606000010.tif68 TIFF2025509606000011.tif34 shows the amplitude coefficients with TIFF2025509606000012.tif611 shows recursive averaging over time to perform smoothing of the subband signals.

[0025] According to an embodiment, the signal processor 120 may be configured to generate a preprocessed audio signal, for example by dereverberating the audio input signal to attenuate original reverberant signal components of the audio input signal, and to obtain an audio output signal depending on the reverberation gain information, for example by adding artificial reverberation to the preprocessed audio signal.

[0026] In one embodiment, the signal processor 120 may be configured to generate a pre-processed audio signal, for example by performing a temporal smoothing of the audio input signal. The signal processor 120 may be configured to obtain an audio output signal depending on the reverberation gain information, for example by adding artificial reverberation to the pre-processed audio signal.

[0027] According to one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the audio input signal, for example.

[0028] In one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the audio input signal, for example.

[0029] According to one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the volume of the audio input signal, for example.

[0030] In one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the variance of one or more features of the audio input signal, for example.

[0031] According to an embodiment, the device may comprise, for example, a ring buffer with a long length and / or a long overlap, for example for real-time processing, which may be configured to receive, for example, an audio input signal or a pre-processed audio signal. The signal processor 120 may be configured to process, for example, the audio input signal or the pre-processed audio signal in the ring buffer to obtain an audio output signal.

[0032] In the following, specific embodiments are described. First, an overview can be presented, for example, presenting an algorithm according to one embodiment, then results including listening tests and statistical evaluation are provided. Furthermore, a model for predicting the perceived reverberation intensity according to one embodiment is provided, post-processing using a scaling factor according to one embodiment is presented, real-time application of the trained model according to one embodiment is explained, and conversion to reverberation transmit gain is presented.

[0033] In some embodiments, the perceived intensity of reverberation in an audio signal can be estimated, for example, and the level of the artificial reverberation signal can be controlled, for example, so that the artificially reverberated output signal has similar reverberation characteristics, for example, as the corresponding input signal. The estimation can employ, for example, a linear regression model with subband inter-channel coherence and spectral flatness measures as input features, trained on listening test data. For adaptation (for example, for application of a scaling factor), the artificial reverberation control signal can be calculated, for example, depending on the time modulation characteristics and / or depending on, for example, the correlation between the input channel signals, and can be applied, for example, to calculate an equivalent reverberation level. The resulting quantities can be post-processed, for example, using signal adaptive integration. These concepts can be applied, for example, to control the reverberation transmit gain of an artificial reverberation used for sound reproduction in a car (reverberation transmit gain: for example, the (transmit) gain of the (artificial) reverberation).

[0034] Reverberation is a very complex element in acoustics and audio signal processing, and embodiments focus on the perceived intensity and its control, for example by adjusting the reverberation transmit gain. Some embodiments may, for example, be implemented independently of other aspects, such as the frequency-dependent reverberation time of the impulse response or its correlation across channels. Other embodiments may, for example, take these aspects into account.

[0035] A schematic diagram of a particular embodiment is shown in Fig. 2. The first stage may include, for example, dereverberation and extraction of audio features. In particular, Fig. 2 shows a block diagram of a reverberation control algorithm according to one embodiment. The feature values ​​are input to a model for predicting the perceived reverberation strength in the input signal. The model uses linear regression and is trained on listening test data. The model output is post-processed with signal-adaptive scaling coefficients that are manually adjusted with respect to the characteristics of the newly added artificial reverberation to address the changed interaction between the excitation signal and the reverberation due to the different characteristics of the artificial reverberation and the "original" reverberation. It is noted that data-driven modeling of the perceived reverberation strength for the artificial reverberation (as is done for the original reverberation) requires listening test data for multiple parameter settings and cannot be adapted when these settings are changed. Therefore, such an approach is not preferred.

[0036] The equivalent reverberation level is post-processed with real-time signal-adaptive time integration. The result is applied to calculate a reverberation transmit gain using a function fitted to the preferred transmit gain tuned by expert listeners. The calculated reverberation transmit gain controls the level of artificial reverberation applied to the dereverberated input signal. Dereverberation is applied to attenuate the input reverberant signal components, because adding artificial reverberation to an input signal with a large amount of "original" reverberation can result in loss of intelligibility, a perception of excessive reverberation, or an aesthetically undesirable superposition of the two reverberant signals.

[0037] The results of the listening test are shown below. In the listening test, 27 volume-normalized audio signals with durations ranging from 4.8 to 14.7 seconds were presented. Items were selected to reveal only small variations in spatial cues (amount of reverberation and energy distribution in the stereo panorama). The item set ranged from very dry to very reverberant and included recordings of various music genres, as well as recordings of isolated instruments and dry speech.

[0038] Participants were asked to rate the perceived reverberation intensity and ensemble width by adjusting sliders on separate unipolar scales ranging from 1 to 9. The labels "very low", "low", "medium", "high", and "very high" were uniformly distributed on the scale. An additional attribute, ensemble width, was provided to help listeners judge the reverberation intensity independently of the stereo panorama. The test begins with a training session in which three stimuli are presented to give examples of, for example, low amounts of reverberation and wide ensemble width, and vice versa. Fifteen listeners participated in the test.

[0039] FIG. 3 illustrates the average ratings per listening test item and the grand mean with 95% confidence intervals, according to one embodiment. In particular, FIG. 3 illustrates the averages and 95% confidence intervals for the reverberation ratings. The confidence intervals are: TIFF2025509606000013.tif1746(1) It was obtained according to, where TIFF2025509606000014.tif64m is the estimated average TIFF2025509606000015.tif619, TIFF2025509606000016.tif58 is from the t direction TIFF2025509606000017.tif4 Quantiles with 11 degrees of freedom (here, TIFF2025509606000018.tif33=0.05), TIFF2025509606000019.tif54 is the standard deviation of the estimate, TIFF2025509606000020.tif33 is a sample size [8].

[0040] Below is a linear regression model according to one embodiment. The following describes a computational model for predicting the perceived reverberation strength of an audio signal and its training according to certain embodiments.

[0041] First, a model description according to one embodiment is provided. The linear regression model predicts the perceived reverberation intensity. TIFF2025509606000021.tif53, input features Linear combination of TIFF2025509606000022.tif56, TIFF2025509606000023.tif2152(2) where the model coefficients are TIFF2025509606000024.tif66, bias value is TIFF2025509606000025.tif66, the number of input features is The file is TIFF2025509606000026.tif44.

[0042] Next, audio feature extraction according to the embodiment will be described. The inputs to the linear regression model are the subband inter-channel coherence (ICC) and the spectral flatness measure (SFM) calculated from the short-term Fourier transform (STFT) coefficients. As shown in Table 1, the ICC is calculated in 5 and the SFM is calculated in 4 frequency bands.

[0043] Table 1 shows the frequency band separation for ICC and SFM according to one embodiment.

[0044] [Table 1]

[0045] The STFT coefficients are calculated from an audio signal sampled at 48 kHz, with a frame size of 1024 samples, a hop length of 512 samples, and no zero padding. STFT frames with a volume below -65 LUFS (loudness units relative to full scale) were removed to make the model training independent of periods of silence, e.g., speech pauses. ICC [9] states that TIFF2025509606000028.tif21101(3) The input signal is calculated from both channel signals of a two-channel stereo input signal according to TIFF2025509606000029.tif724 and TIFF2025509606000030.tif724 are the automatic power spectral density (PSD) of the left and right channels, respectively. TIFF2025509606000031.tif724 is the time index In TIFF2025509606000032.tif34 TIFF2025509606000033.tif is the cross PSD where all PSDs are accumulated over the frequency bin corresponding to the 54th frequency band. For example, TIFF2025509606000034.tif618 is the first audio channel of the audio input signal. TIFF2025509606000035.tif54th frequency band TIFF2025509606000036.tif shows the 42nd complex spectrum bin, TIFF2025509606000037.tif618 is the second audio channel of the audio input signal. TIFF2025509606000038.tif54th frequency band TIFF2025509606000039.tif4Let's say it indicates the 2nd complex spectrum bin. TIFF2025509606000040.tif724, TIFF2025509606000041.tif724, and For example, TIFF2025509606000042.tif724 is TIFF2025509606000043.tif1572TIFF2025509606000044.tif1575TIFF2025509606000045.tif1572, where the time index is As TIFF2025509606000046.tif34, TIFF2025509606000047.tif521 TIFF2025509606000048.tif520 shows the complex conjugate, TIFF2025509606000049.tif521 Shows the complex conjugate of TIFF2025509606000050.tif519. TIFF2025509606000051.tif44, for example, contains the frequency band Frequency bins in TIFF2025509606000052.tif43 The number of downmix squared STFT magnitude coefficients for TIFF2025509606000053.tif55 can be shown.

[0046] For example, in certain embodiments, a frequency band may include, for example, one or more frequency bins per audio channel (e.g., in certain embodiments, a frequency band may simply be, for example, a single bin). For example, in certain embodiments, Equation (3) may be employed, for example, to obtain inter-channel correlation for such frequency bands.

[0047] SFM (see

[10] ) TIFF2025509606000054.tif26104(4) It is calculated as the ratio of the geometric mean to the arithmetic mean according to TIFF2025509606000055.tif13100(5) where: TIFF2025509606000056.tif610 is the compression function, e.g. logarithm, or square root, or TIFF2025509606000057.tif723, TIFF2025509606000058.tif929 is the time index TIFF2025509606000059.tif34, TIFF2025509606000060.tifFrequency band consisting of 44 bins In TIFF2025509606000061.tif43 TIFF2025509606000062.tif 42nd frequency bin are the downmix squared STFT magnitude coefficients of the left and right input channels of TIFF2025509606000063.tif68. (Hence, TIFF2025509606000064.tif44, for example, contains the frequency band Frequency bins in TIFF2025509606000065.tif43 TIFF2025509606000066.tif68.) For example, in an embodiment having two audio channels, the downmix squared STFT magnitude coefficients TIFF2025509606000067.tif929 is, for example, the downmix STFT amplitude coefficients It may be the result of squaring TIFF2025509606000068.tif827, downmix STFT amplitude coefficients For example, TIFF2025509606000069.tif827 is TIFF2025509606000070.tif5 It may also be the result of combining (e.g. averaging) the amplitude coefficients of two audio channels of a frequency bin of the 4th frequency band.

[0048] In an embodiment, features were extracted for each frame (eg, for each block) and then the arithmetic mean and standard deviation were calculated to obtain a single value per block, which was used to train the regression model.

[0049] For example, each block may contain audio data for, for example, playing one second of the recording. A block may contain, for example, spectral data for 94 time points (e.g., for 94 frames, assuming that, for example, in a particular embodiment, the frames have a duration of, for example, 21.3 milliseconds, with 50% overlap of the frames, e.g., approximately 94 frames / second). In other embodiments, a block may contain, for example, spectral data for any other number of time points. In one example, the spectrum may be divided into, for example, five frequency bands and / or, for example, four frequency bands, e.g., as outlined in Table 1.

[0050] For example, taking ICC into consideration, an ICC value is determined for each of, for example, five frequency bands in table 1 and for each of, for example, 94 time points. For example, for each of the five frequency bands, an arithmetic mean of, for example, 94 ICCs of the block is determined, thus, for example, five arithmetic mean ICC values ​​of the five frequency bands may result. And / or, for example, for each of the five frequency bands, a standard deviation of, for example, 94 ICCs of the block is determined, thus, for example, five standard deviation ICC values ​​of the five frequency bands may result. This results in, for example, 10 ICC-related feature values ​​for, for example, five frequency bands of the block.

[0051] For example, taking into account the SFM, an SFM value is determined for each of, for example, four frequency bands in Table 1 and for each of, for example, 94 time points. For example, for each of the four frequency bands, an arithmetic mean of, for example, 94 SFMs of the block is determined, thus resulting in, for example, four arithmetic mean SFM values ​​for the four frequency bands. And / or, for example, for each of the four frequency bands, a standard deviation of, for example, 94 ICCs of the block is determined, thus resulting in, for example, four standard deviation SFM values ​​for the four frequency bands. This results in, for example, eight SFM-related feature values ​​for, for example, four frequency bands of the block.

[0052] For a combination of both ICC and SFM examples, for example, 18 feature values ​​for the block may be obtained.

[0053] Next, model training according to the embodiment will be described. Processing short blocks of data is essential for real-time processing, especially for non-stationary signals, and to increase the available amount of training data. Each item was windowed with a training block length of 6 seconds with 0 overlap. Items shorter than 12 seconds in length were treated as a single block for their entire length. Performance when training with overlapping blocks was also evaluated, and no improvement in model accuracy was observed.

[0054] The training dataset was extended by adding a second excerpt with similar spatial characteristics for some of the songs used in the listening tests. Furthermore, several additional important items that were poorly predicted were added along with reference ratings provided by one expert listener. To ensure that the model predicts small values ​​of dry and clean speech signals, recordings of male and female speakers were added to the dataset with reference annotations set to -1 MOS (mean opinion score). Expanding the dataset and using multiple blocks for each item resulted in a total of 100 observations.

[0055] To gain the opportunity to account for non-linear relationships between the listening test data and the feature values, an index was applied to the mean and standard deviation of the feature values. The index was determined by manual tuning based on evaluation of residual plots, which by definition reveal non-linearities that cannot be modeled by linear regression.

[0056] Training a model is a process of predicting This was performed with the ordinary least squares (OLS) algorithm, which minimizes the mean squared error between the mean ratings obtained in the listening test and the mean ratings obtained in the listening test. The evaluation was performed using leave-one-out cross-validation

[11] , where the model is trained on all observations except one and evaluated on the left-out item. This procedure is repeated for each item.

[0057] Next, feature selection according to the embodiment will be described. Training started with all available values, e.g., the arithmetic mean and standard deviation for each subband feature, which could be e.g., the ICC or SFM for the aforementioned subbands (frequency bands).

[0058] To avoid overfitting and reduce the computational load, we removed non-informative independent variables one by one by evaluating their p-values, where a low p-value indicates most likely no relationship between the corresponding feature value and the listening test results (see

[11] ).

[0059] For example, in the above example with 18 feature values ​​(10 ICC-related feature values ​​and 8 SFM-related feature values), gradually removing all irrelevant feature values ​​may result in a model with e.g. 4 input variables (e.g. 14 other irrelevant or less relevant input variables / feature values ​​may be gradually removed, e.g. by performing a multivariate regression analysis (see, e.g.,

[11] ). The performance of this model is shown in Figure 4.

[0060] In particular, Figure 4 shows predicted reverberation strengths against observed reverberation strengths according to one embodiment. Predictions for training blocks of listening test items are displayed as dots, while predictions obtained for training blocks of the extended data set (items from the extended data set) are represented as triangles. The average prediction per item is shown as a cross. Markers representing blocks of the same items have the same listening test rating. The plot reveals a good correlation between predicted and observed, with a correlation coefficient of 0.75 and a mean error of 1.27 MOS. For listening test ratings around 3MOS, there is a trend towards too high values, while high ratings are estimated slightly under.

[0061] In the following, a reverberation transmit gain calculation according to one embodiment is provided. In particular, a calculation of a preferred reverberation transmit gain of an artificial reverberation is described, given the perceived intensity of the first-order reverberation in the input signal. Since the original reverberation (of the input signal) and the second-order (artificial) reverberation do not match, equal reverberation levels can result in different perceived reverberation intensities. Therefore, the reverberation transmit gain cannot be calculated directly, taking into account the perceived reverberation intensity.

[0062] Data-driven modeling of the perceived strength of artificial reverberation is not feasible because it requires subjective data for different reverberation settings.

[0063] In an embodiment, manually adjusted scaling factors TIFF2025509606000072.tif55 and TIFF2025509606000073.tif55 is introduced to compensate for signal-dependent effects, for example as explained below. For the purposes of the processing description, the equivalent reverberation level The intermediate quantity, TIFF2025509606000074.tif64, is defined as follows: TIFF2025509606000075.tif1148(6)

[0064] The equivalent reverberation level represents the desired level of artificial reverberation for a given input signal that results in similar reverberation strength of the artificial and original reverberation. The equivalent reverberation level is then converted to a reverberation transmit gain using a mapping function determined in a separate tuning experiment.

[0065] Next, an aspect of the equivalent reverberation level according to the embodiment will be described. Signals with strong transients, e.g. drum sounds, result in a higher perceived reverberation intensity than stationary signals and are less effective at masking reverberation.

[0066] Artificial reverberation with large time constants may require lower reverberation levels to evoke a similar intensity perception as the original reverberation. Some embodiments provide a novel Spectral Transient Measure (STM) to quantify the intensity of transients in a signal as follows: TIFF2025509606000076.tif14132(7) During the ceremony, TIFF2025509606000077.tif2182(8) It is. TIFF2025509606000078.tif611 shows recursive averaging over time to perform smoothing of the subband signals.

[0067] For example, in certain embodiments, the frequency bands may be, for example, the frequency bands in the left column of Table 1 or the frequency bands in the right column of Table 1. Alternatively, other frequency bands / frequency band configurations may be used. Each of the frequency bands may then be, for example, a time index. For TIFF2025509606000079.tif34, all time-frequency bins within the aforementioned frequency bands may be included.

[0068] The STM single value is obtained by averaging the STM values ​​in the frequency bands 23-70 Hz and 5.2-21 kHz. This band selection focuses on the transients caused by percussion instruments, while the transients of other instruments, e.g. the piano, are not taken into account as much.

[0069] Scaling Factor TIFF2025509606000080.tif55 is calculated from the STM as follows: TIFF2025509606000081.tif1192(9) Where: TIFF2025509606000082.tif512 shows the average STM values ​​of a 1 second long block which was then cropped according to: TIFF2025509606000083.tif1261(10) TIFF2025509606000084.tif46, TIFF2025509606000085.tif44, TIFF2025509606000086.tif411, and TIFF2025509606000087.tif411 are adjustable parameters that are adjusted depending on the characteristics of the artificial reverberation.

[0070] If the reverberation time of the "original" reverberation exceeds the one of the artificial reverberation, a larger equivalent reverberation level is required. The second scaling factor TIFF2025509606000088.tif55 is designed to scale the estimated reverberation intensity of input signals that contain a large amount of reverberation with long reverberation times, e.g. classical music. Such items are assumed to be fairly uncorrelated due to the large amount of diffuse sound content. TIFF2025509606000089.tif55 is calculated as follows: TIFF2025509606000090.tif1158(11) Where: TIFF2025509606000091.tif510 is the average ICC of all frequency bands and frames within a 1 second long block, which is later cropped according to: TIFF2025509606000092.tif1057(12) During the ceremony, TIFF2025509606000093.tif46, TIFF2025509606000094.tif45, TIFF2025509606000095.tif411, and TIFF2025509606000096.tif412 is an adjustable parameter.

[0071] In the following, signal adaptive time integration according to an embodiment will be described. To control the reverberation transmit gain in real time, it is desirable to react to changes in the input signal without introducing noticeable modulation of the reverberation level. To address this with low latency and low computational load, predictions are calculated from blocks of 8 seconds length, each with an overlap of 7 seconds, such that new predictions are calculated at a rate of 1 Hz. Signals with low input levels, e.g. below -65 LUFS, are not fed to the model as was done during training. Equivalent reverberation level TIFF2025509606000097.tif613(6) is smoothed in time using recursive averaging with a single-pole IIR filter to get: TIFF2025509606000098.tif10108(13) Here, the smoothing coefficient is Let's say TIFF2025509606000099.tif34.

[0072] To avoid too large reverberation levels at the transitions between input signals with very different characteristics, track changes are detected and a fast adaptation of the equivalent reverberation level is performed. The fast adaptation reduces the model block length to 1 second and reduces the smoothing factor This is done by increasing the TIFF2025509606000100.tif34. For detection, a change in the variance of the STM is detected, the reason for this is that a change in the amount of transients is a good indicator for a transition from e.g. music to speech, which requires a much lower reverberation level.

[0073] Figure 5 shows real-time processing of an illustrative input file consisting of 25 seconds of classical music and an abrupt transition to pop music, which can be observed in the top plot showing a spectrogram according to an embodiment. In particular, the top part of Figure 5 shows a spectrogram of an audio signal that transitions from classical music to pop music after 25 seconds. The middle part of Figure 5 shows the equivalent reverberation level before (solid line) and after (dashed line) temporal smoothing. The bottom part of Figure 5 shows the smoothing factor (solid line) and the total scaling value (dashed line).

[0074] The middle subplot shows the equivalent reverberation levels before (solid line) and after (dashed line) temporal smoothing with values ​​of about 7MOS for classical music and 2.2MOS for pop items. The lower subplot shows the total scaling values The product of the scaling values, called TIFF2025509606000101.tif619, is shown as a dashed line. Due to the low correlation of the classical music items, the total scaling value is higher than 1 until the track changes. An increase in the amount of transients at t=25 s triggers track change detection, reducing the total scaling value to approximately 0.7.

[0075] Smoothing coefficients plotted as solid lines in the lower subplot TIFF2025509606000102.tif34 shows that after an 11 second adjustment phase, the volume and equivalent reverberation level are reduced to a rather stationary state. Track change detection at t=25 seconds TIFF2025509606000103.tif34 is temporarily increased, resulting in a rapid adaptation of the equivalent reverberation level.

[0076] Next, conversion from reverberation prediction to reverberation transmit gain according to an embodiment will be described. For example, as a final step the transmission gain of the reverberation processor may be calculated, for example taking into account the equivalent reverberation level. For this purpose, a second listening test has been carried out, in which five expert listeners adjusted according to their preferences the transmission gain of two artificial reverberations simulating the acoustic environments of a concert hall (T60=2.2 seconds) and a jazz club (T60=1.3 seconds) in a car.

[0077] FIG. 6 illustrates the reference reverberation transmit gain over the equivalent reverberation level with fitted polynomials according to one embodiment. In particular, FIG. 6 shows the obtained average transmit gain over the equivalent reverberation level with 95% confidence interval according to equation (1) for all 30 test items of the two artificial reverberations. Since a longer reverberation tail results in a higher perceived reverberation intensity with less masking and transients requiring lower reverberation transmit gain, the adjustable parameters of equations (9) and (11) were adjusted such that a stronger transient and lower correlation results in a lower equivalent reverberation level for the concert hall compared to the jazz club. As a result, one single mapping curve can be used for both reverberations. With a value of 0.81, the Spearman rank correlation coefficient confirms the monotonic relationship between the two variables that can be approximated with a third order polynomial with low error.

[0078] Using gain adjustment, polynomial mapping can be done as shown in Figure 6. TIFF2025509606000104.tif652 was fitted. This resulted in a smoothed equivalent reverberation level. TIFF2025509606000105.tif66 Predicted reverberation transmit gain Convert to TIFF2025509606000106.tif43.

[0079] The following considers the evaluation mode. To evaluate the polynomial mapping, the technique of leave-one-out cross-validation was used, which means that all pairs of values ​​except one were used to fit the polynomial function, and the last data pair served to obtain the error for this particular data point. This process was repeated for all listening test items with a mean absolute error (MAE) of 1.74 dB. According to [7], such a deviation from the preferred reverberation transmission gain results in a negligible decrease in aesthetic quality. The limits of the 95% confidence interval obtained according to formula (1) are 1.27 dB and 2.21 dB. The absolute maximum error corresponds to 5.11 dB, and the correlation between the predicted and observed values ​​is at 0.89.

[0080] 7 is a diagram illustrating reverberation transmit gain error versus equivalent reverberation level according to one embodiment. The scatter plot shown in FIG. 7 shows the reverberation transmit gain error across equivalent reverberation levels, showing an even distribution of error over the entire range with a single outlier at 2.6MOS.

[0081] Concepts are provided for controlling the level of artificial reverberation. Some of the provided concepts can employ a model for predicting the perceived reverberation intensity based on, for example, a linear regression model trained on listening test data. The method uses manually tuned adaptation to various parameter settings of the artificial reverberation. Fast adaptation to changes in the input signal characteristics is implemented by signal adaptive time integration. The algorithm was evaluated in an in-car sound reproduction application. It is able to predict the preferred reverberation transmit gain with an MAE of 1.74 dB.

[0082] Next, an extension of the embodiment provided above will be described. For example, other application scenarios may be realized, for example.

[0083] The embodiments may be applied, for example, to control reverberation in various other application scenarios: in binaural sound reproduction, for example, artificial reverberation is applied to support a sense of externalization, where the listener localizes acoustic events, objects, and sound sources outside the head, in contrast to standard sound reproduction with headphones, where sounds are usually perceived as coming from inside the head.

[0084] Artificial reverberation is used in music production where it is applied to the mixing of multiple sound sources and individual sound sources. The proposed method can also be extended to predict the amount of perceived reverberation for individual sound sources or individual components of a mixed signal (e.g. groups of sound sources). Information on the amount per individual component allows automatic control or artificial reverberation in audio production (music production, film audio, podcasts, and other user-generated content).

[0085] Other embodiments may, for example, implement alternative models. Linear models can be used to model other data-driven models, e.g., arbitrary functions may be replaced by TIFF2025509606000107.tif934, where scalar output TIFF2025509606000108.tif64 is the input vector TIFF2025509606000109.tif34, and TIFF2025509606000110.tif64 and target output The model parameters are determined by optimizing a criterion between the Given TIFF2025509606000112.tif65, it is computed. Another well-known class of such trainable models are artificial neural networks, in particular deep neural networks (DNNs). The parameters The larger the set of TIFF2025509606000113.tif65, the more data is needed to train the model, and the performance is not only good for the training data, but also for new data pairs that were not used during training. It should be noted that the model is also good for TIFF2025509606000114.tif613. This property is called generalization. However, with more training data available, other models such as DNNs can yield better predictions (with higher accuracy).

[0086] Such DNNs are implemented by combining layers of processing units, where the units themselves have trainable parameters. Commonly used types of layers are convolutional layers, dense layers (also called fully connected layers) and recurrent layers. They differ in what types of units are implemented and how these units are connected to each other. Additional types of layers are used to support the training process, where training is the process of training the parameters by numerical optimization. Refers to the process of optimizing TIFF2025509606000115.tif65.

[0087] Further embodiments are provided below. An apparatus and / or method is provided for controlling the level of an artificially generated reverberant signal added to an audio signal using a model of the perceived intensity of reverberation in the original input signal.

[0088] According to one embodiment, the input signal may be dereverberated, for example, to attenuate the original reverberant signal components.

[0089] In one embodiment, a linear regression model may be employed, for example, with different audio feature values ​​as input.

[0090] According to one embodiment, sub-band SFM and / or sub-band ICC may, for example, be employed as input audio features.

[0091] In one embodiment, an adjustable scaling value for modifying the model prediction may for example be employed, which depends on the audio feature values, to compensate for the altered interaction of the artificial reverberation and the dereverberated input signal compared to the original reverberation and direct signals.

[0092] According to one embodiment, an adjustable scaling value that depends, for example, on the subband ICC and the subband STM may be employed.

[0093] In one embodiment, a ring buffer with high overlap for real-time processing may be employed, for example.

[0094] According to one embodiment, adaptive temporal smoothing may be employed, for example, to adjust the amount of smoothing according to changes in the input signal.

[0095] In one embodiment, the changes in loudness and model predictions may be evaluated to control, for example, a smoothing factor for temporal recursive averaging.

[0096] According to one embodiment, a track change may be detected in response to a change in the variance of the audio features, for example to temporarily reduce a smoothing factor of the temporal recursive averaging and reset the ring buffer.

[0097] In one embodiment, a mapping function fitted to the listening test data can be used, for example, to convert the conditioned model predictions into reverberation transmit gains.

[0098] The proposed concept can be applied, for example, to sound reproduction in automobiles to mimic an acoustic environment of larger size and pleasant sound spatial characteristics. The emulation of the acoustic environment is achieved by processing the audio input signal such that the reverberant signal components of the output signal are perceptually similar to the reproduction of the direct signal components in the new environment. This implementation has low latency and low workload.

[0099] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0100] Depending on the specific implementation requirements, embodiments of the present invention can be implemented in hardware or software, or at least partly in hardware, or at least partly in software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to execute the respective method. The digital storage medium may therefore be computer readable.

[0101] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0102] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer. The program code can, for example, be stored on a machine readable carrier.

[0103] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0104] In other words, therefore, an embodiment of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0105] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0106] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0107] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0108] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0109] Further embodiments according to the invention comprise an apparatus or a system arranged to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0110] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0111] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0112] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0113] The above-described embodiments are merely illustrative of the principles of the present disclosure. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and explanations of the embodiments herein.

[0114] References [1] Paulus, J., Uhle, C., and Herre, J., “Perceived Level of Late Reverberation in Speech and Music,” in Proc. AES 130th Conv., 2011. [2] Gardner, W. G. and Griesinger, D., “Reverberation level matching experiments,” in Proc. Sabine Centennial Symposium, Acoust. Soc. of Am., 1994. [3] Leonard, B., King, R., and Sikora, G., “The Effect of Playback System on Reverberation Level Preference,” in Audio Engineering Society Convention 134, 2013. [4] Leonard, B., King, R., and Sikora, G., “The Effect of Acoustic Environment on Reverberation Level Preference,” in Audio Engineering Society Convention 133, 2012. [5] Sarroff, A. M. and Bello, J. P., “Toward a Computational Model of Perceived Spaciousness in Recorded Music,” J. Audio Eng. Soc, 59(7 / 8), pp. 498-513, 2011. [6] Uhle, C., Paulus, J., and Herre, J., “Predicting the Perceived Level of Late Reverberation Using Computational Models of Loudness,” in Proc. 17th Int. Conf. on Digital Signal Process. (DSP), 2011. [7] Paulus, J., Uhle, C., Herre, J., and Hoepfel, M., “A Study on the Preferred Level of Late Reverberation in Speech and Music,” in Proc. of the 60th Int. Conf. of Audio Eng. Soc., 2016. [8] Gilleland, E., Confidence Intervals for Forecast Verification, National Center For Atmospheric Research, Boulder, Colorado, 2010. [9] Allen, J., Berkeley, D., and Blauert, J., “Multimicrophone Signal-Processing Technique to Remove Room Reverberation from Speech Signals,” J. Acoust. Soc. Am., 62, 1977.

[10] Gray, A. H. and Markel, J. D., “A Spectral-Flatness Measure for Studying the Autocorrelation Method of Linear Prediction of Speech Analysis,” IEEE Trans. Acoust., Speech, and Sig. Process., 22, pp. 207-217, 1974.

[11] James, G., Witten, D., Hastie, T., and Tibshirani R., editors, “An Introduction to Statistical Learning: with Applications in R,” number 103 in Springer Texts in Statistics, Springer, New York, 2013.

Claims

1. 1. An apparatus for processing an audio input signal comprising one or more audio channels to obtain an audio output signal, said apparatus comprising: a reverberation gain determiner (110) configured to determine reverberation gain information in response to the audio input signal; and a signal processor (120) configured to obtain the audio output signal in response to the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

2. The apparatus of claim 1 , wherein the reverberation gain determiner (110) is configured to determine the reverberation gain information in response to an estimate of a perceived intensity of reverberation in the audio input signal.

3. 3. The apparatus of claim 2, wherein the reverberation gain determiner (110) is configured to determine the reverberation gain information by employing a model that, upon receiving information regarding one or more features of the audio input signal, returns the estimate of the perceived intensity of the reverberation in the audio input signal.

4. 4. The apparatus of claim 3, wherein the model employed by the reverberation gain determiner (110) is a linear regression model using one or more feature values ​​of the one or more features of the audio input signal as input for the linear regression model.

5. 5. The apparatus of claim 3, wherein to obtain a reverberation transmit gain, the reverberation gain determiner (110) is configured to determine the reverberation transmit gain as the reverberation gain information by mapping the estimate of the perceived intensity of the reverberation in the audio input signal according to a mapping function.

6. 6. The apparatus of claim 3, wherein the one or more features of the audio input signal depend on inter-channel correlation of at least one of one or more subbands of two audio channels of the one or more audio channels of the audio input signal.

7. 7. The apparatus of claim 3, wherein the one or more features of the audio input signal depend on at least one spectral flatness measure of one or more subbands of the one or more audio channels of the audio input signal.

8. the reverberation gain determiner is configured to determine the estimate of the perceived intensity of the reverberation in the audio input signal by using the model; the reverberation gain determiner is configured to determine one or more scaling factors in response to the one or more features of the audio input signal; 8. The apparatus of claim 3, wherein the reverberation gain determiner is configured to determine the reverberation gain information in response to the estimate of the perceived reverberation intensity and in response to the one or more scaling factors.

9. 9. The apparatus of claim 8, wherein the one or more scaling factors depend on an inter-channel correlation of at least one of one or more sub-bands of two audio channels of the one or more audio channels of the audio input signal.

10. 10. Apparatus according to claim 8 or 9, wherein the one or more scaling factors depend on the presence of transient signal components in at least one of the one or more audio channels of the audio input signal.

11. the one or more scaling factors being dependent on a spectral transient measurement in at least one of the one or more audio channels of the audio input signal; The spectral transient measurements include: is defined according to the formula: and indicates the time index, denotes the frequency index, is the frequency band / subband , where a frequency band of at least one of the one or more audio channels or a combination of the at least one of the one or more audio channels The one or more frequency bins in Time index for denotes an amplitude coefficient having 11. The device according to claim 8, wherein x denotes a recursive averaging over time to effect smoothing of the subband signals.

12. the signal processor (120) is configured to generate the preprocessed audio signal by dereverberating the audio input signal to attenuate original reverberant signal components of the audio input signal; 12. The apparatus of claim 1, wherein the signal processor (120) is configured to obtain the audio output signal in response to the reverberation gain information by adding the artificial reverberation to the pre-processed audio signal.

13. the signal processor (120) is configured to generate the pre-processed audio signal by performing temporal smoothing of the audio input signal; 13. The apparatus of claim 1, wherein the signal processor (120) is configured to obtain the audio output signal in response to the reverberation gain information by adding the artificial reverberation to the pre-processed audio signal.

14. The apparatus of claim 13 , wherein the signal processor (120) is configured to adjust the amount of temporal smoothing in response to changes in the audio input signal.

15. 15. Apparatus according to claim 13 or 14, wherein the signal processor (120) is configured to adjust the amount of temporal smoothing in response to changes in the audio input signal.

16. The apparatus of claim 15, wherein the signal processor (120) is configured to adjust the amount of temporal smoothing in response to changes in volume of the audio input signal.

17. 17. Apparatus according to claim 15 or 16 further dependent on claim 3, wherein the signal processor (120) is configured to adjust the amount of temporal smoothing in response to changes in variance of the one or more features of the audio input signal.

18. the apparatus comprises a ring buffer for receiving the audio input signal or the pre-processed audio signal, 18. The apparatus of claim 1, wherein the signal processor (120) is configured to process the audio input signal or the pre-processed audio signal in the ring buffer to obtain the audio output signal.

19. 1. A method for processing an audio input signal to obtain an audio output signal, said method comprising the steps of: determining reverberation gain information in response to the audio input signal; and obtaining the audio output signal in response to the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

20. 20. A computer program for performing the method according to claim 19 when the computer program is run on a computer or signal processor.

Images