Apparatus and method for automatic control of reverberation levels using a perceptual model

Abstract

According to one embodiment, an apparatus is provided for processing an audio input signal comprising one or more audio channels to obtain an audio output signal. The apparatus comprises a reverberation gain determiner (110) configured to determine reverberation gain information in response to the audio input signal. Furthermore, the apparatus comprises a signal processor (120) configured to obtain the audio output signal in response to the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

Description

Technical Field

[0001] The present invention relates to audio signal processing, and more particularly to an apparatus and method for automatically controlling reverberation levels, such as by using a perceptual model.

Background Art

[0002] Reverberation is a very complex element in acoustics and audio signal processing, and its perceived intensity and its control are particularly important. Audio signals, such as music recordings or radio broadcasts, can be processed, for example, by artificial reverberation to emulate the acoustic characteristics of a particular environment, such as a concert hall or a hall. The input signal may be, for example, a mixture of a direct signal and an ambient signal. The direct signal may be, for example, a recording of singing, musical instruments, and sound events such as gunshots and alarm sirens. The term "direct sound" indicates that these sounds are directional and can be positioned as coming from one direction. The ambient (or diffuse) sound component may be, for example, reverberation and environmental sounds, such as wind noise and rain, that are not perceived as coming from a particular direction. In music recordings, reverberation is the most prominent source of ambient sound.

[0003] The perceived reverberation level depends on the input signal and the reverberation impulse response, such as the pre-delay and the length of the reverberation tail (see [1]). For example, non-stationary input signals with transients and fast attacks of sub-band envelopes, such as drum sounds, generate higher reverberation intensities.

[0004] Input signals with rapidly decaying envelopes and silent portions are not very effective in masking reverberation (see [2]). When the reverberation time and pre-delay are small, the input signal coincides with the reverberation signal and masks the reverberation signal more significantly, as in the case where the reverberation time and pre-delay are large. Furthermore, other aesthetic aspects related to the acoustic environment (see [3]), the playback system (see [4]), and the genre of music affect the preferred gain parameter settings.

[0005] A model for predicting multiple spatial attributes adjusted for listening test data is described in [5], although the proposed model was not applicable in real time, and the model itself was more complex, using up to 12 audio features, while yielding comparable performance to the provided algorithm which uses only three features. (See [6]) The model for perceived levels of late reverberation uses the partial volume of the direct and reverberation components of the input audio signal. A study in [1] on perceived levels of artificial late reverberation showed that perceived levels depend on reverberation time and input signal, but not on inter-channel correlation of impulse response, and that there were similar differences in evaluation between individuals and in repeated tests of the same individual. Preferred levels have been investigated in [7], and it has been concluded that aesthetic quality deteriorates more when the applied reverberation is above the preferred level. [Overview of the project] [Problems that the invention aims to solve]

[0006] The object of the present invention is to provide an improved concept for automatically controlling reverberation levels. The object of the present invention is solved by the apparatus described in claim 1, the method described in claim 19, and the computer program described in claim 20. [Means for solving the problem]

[0007] An apparatus is provided for processing an audio input signal including one or more audio channels in order to acquire an audio output signal, according to one embodiment. The apparatus includes a reverberation gain determiner configured to determine reverberation gain information in accordance with the audio input signal. Furthermore, the apparatus includes a signal processor configured to acquire an audio output signal in accordance with the reverberation gain information by adding artificial reverberation to the audio input signal or a pre-processed audio signal dependent on the audio input signal.

[0008] Furthermore, a method for processing an audio input signal to acquire an audio output signal is provided according to one embodiment. This method is - Determining reverberation gain information according to the audio input signal, -By adding artificial reverberation to the audio input signal or a pre-processed audio signal that depends on the audio input signal, an audio output signal is obtained according to the reverberation gain information, It includes.

[0009] Furthermore, a computer program is provided for carrying out the above-described method when it is executed on a computer or signal processor.

[0010] Embodiments of the present invention will be described in more detail below with reference to the drawings. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an apparatus for processing an audio input signal, which includes one or more audio channels, in order to acquire an audio output signal, according to one embodiment. [Figure 2] This is a block diagram of a reverberation control algorithm according to one embodiment. [Figure 3] This figure shows the average evaluation and overall average per listening test item with a 95% confidence interval according to one embodiment. [Figure 4] This figure shows the predicted reverberation intensity relative to the observed reverberation intensity, according to one embodiment. [Figure 5] The top of Figure 5 shows the spectrogram of the audio signal transitioning from classical music to pop music after 25 seconds; the middle of Figure 5 shows the equivalent reverberation level before and after temporal smoothing; and the bottom of Figure 5 shows the smoothing coefficient and total scaling value according to one embodiment. [Figure 6] This figure shows a reference reverberation transmission gain over equivalent reverberation levels having a fitted polynomial, according to one embodiment. [Figure 7]This figure shows the equivalent reverberation level with respect to the reverberation transmission gain error according to one embodiment. [Modes for carrying out the invention]

[0012] Figure 1 shows an apparatus for processing an audio input signal, which includes one or more audio channels, in order to acquire an audio output signal, according to one embodiment.

[0013] This device includes a reverberation gain determination unit 110 configured to determine reverberation gain information in response to an audio input signal.

[0014] Furthermore, the device includes a signal processor 120 configured to acquire an audio output signal according to reverberation gain information by adding artificial reverberation to an audio input signal or a pre-processed audio signal dependent on the audio input signal.

[0015] According to one embodiment, the reverberation gain determination device 110 may be configured to determine reverberation gain information in accordance with an estimation of the perceived intensity of reverberation in an audio input signal.

[0016] In one embodiment, the reverberation gain determination device 110 may be configured to determine reverberation gain information by, for example, receiving information about one or more features of an audio input signal and then employing a model that returns an estimate of the perceived intensity of reverberation in the audio input signal.

[0017] According to one embodiment, the model employed by the reverberation gain determination device 110 may be, for example, a linear regression model, which uses one or more feature values ​​of one or more features of the audio input signal as input for the linear regression model.

[0018] In one embodiment, in order to obtain the reverberation transmission gain, the reverberation gain determiner 110 may be configured to determine the reverberation transmission gain as reverberation gain information, for example, by mapping an estimation of the perceived intensity of reverberation in the audio input signal according to a mapping function. Equal reverberation intensity (= the intensity perceived after application of the scaling value) may be supplied to the mapping function, for example. For example, an estimation of the perceived intensity of reverberation may be mapped to listening test data, and the respectively fitted curves may be used, for example, for subsequent conversion to the reverberation transmission gain.

[0019] According to one embodiment, one or more characteristics of the audio input signal may depend, for example, on the inter-channel correlation of at least one of one or more sub-bands of one or more audio channels of the audio input signal.

[0020] In one embodiment, one or more characteristics of the audio input signal may depend, for example, on the spectral flatness measure of at least one of one or more sub-bands of one or more audio channels of the audio input signal.

[0021] According to one embodiment, the reverberation gain determiner may be configured to determine an estimation of the perceived intensity of reverberation in the audio input signal, for example, by using a model. The reverberation gain determiner may be configured to determine one or more scaling factors according to one or more characteristics of the audio input signal, for example. Further, the reverberation gain determiner may be configured to determine the reverberation gain information according to the estimation of the perceived intensity of reverberation and according to one or more scaling factors, for example.

[0022] In one embodiment, one or more scaling factors may depend on, for example, the inter-channel correlation of at least one of one or more sub-bands of two audio channels of one or more audio channels of an audio input signal.

[0023] According to one embodiment, one or more scaling factors may depend on, for example, the presence of transient signal components in at least one of one or more audio channels of an audio input signal.

[0024] In one embodiment, one or more scaling factors may depend on, for example, spectral transient measurements in at least one of one or more audio channels of an audio input signal. The spectral transient measurements may be defined, for example, in accordance with TIFF0007872853000001.tif14132, where TIFF0007872853000002.tif2182, [[ID=M13]] TIFF0007872853000003.tif34 indicates the time index, TIFF0007872853000004.tif54 indicates the frequency index, TIFF0007872853000005.tif68 is the frequency band / sub-band TIFF0007872853000006.tif43 indicates one of one or more frequency bins, TIFF0007872853000007.tif827 is the frequency band of at least one of one or more audio channels, or of a combination in at least one of one or more audio channels TIFF0007872853000008.tif43 of one or more frequency bins TIFF0007872853000009.tif68 the time index related to TIFF0007872853000010.tif34 shows the amplitude coefficient having TIFF0007872853000011.tif611 demonstrates time-based recursive averaging and performs smoothing of subband signals.

[0025] According to one embodiment, the signal processor 120 may be configured to generate a pre-processed audio signal by, for example, de-reverberating the audio input signal to attenuate the original reverberation signal component of the audio input signal. The signal processor 120 may also be configured to obtain an audio output signal according to reverberation gain information by, for example, adding artificial reverberation to the pre-processed audio signal.

[0026] In one embodiment, the signal processor 120 may be configured to generate a pre-processed audio signal, for example, by performing temporal smoothing of the audio input signal. The signal processor 120 may also be configured to acquire an audio output signal according to reverberation gain information, for example, by adding artificial reverberation to the pre-processed audio signal.

[0027] According to one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the audio input signal, for example.

[0028] In one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the audio input signal, for example.

[0029] According to one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the volume of the audio input signal, for example.

[0030] In one embodiment, the signal processor 120 may be configured to adjust the amount of temporal smoothing in response to changes in the variance of one or more features of the audio input signal, for example.

[0031] According to one embodiment, the apparatus may include, for example, a ring buffer having a long length and / or long overlap for real-time processing, which may be configured to receive, for example, an audio input signal or a pre-processed audio signal. The signal processor 120 may be configured, for example, to process the audio input signal or pre-processed audio signal in the ring buffer to obtain an audio output signal.

[0032] The following describes specific embodiments. First, an overview of an algorithm according to one embodiment may be presented, followed by results including listening tests and statistical evaluations. Furthermore, a model for predicting perceived reverberation intensity according to one embodiment is provided, post-processing using scaling factors according to one embodiment is presented, real-time application of a trained model according to one embodiment is described, and conversion to reverberation transmission gain is presented.

[0033] In some embodiments, the perceived intensity of reverberation in an audio signal can be estimated, for example, and the level of the artificial reverberation signal can be controlled, for example, so that the artificially reverberated output signal has similar reverberation characteristics to, for example, the corresponding input signal. The estimation can employ, for example, a linear regression model with subband inter-channel coherence and spectral flatness measures as input features trained on listening test data. For adaptation (e.g., for the application of scaling coefficients), the artificial reverberation control signal may be calculated, for example, according to time modulation characteristics and / or, for example, according to the correlation between input channel signals, and may be applied, for example, to calculate the equivalent reverberation level. The resulting quantity can be post-processed, for example, using signal adaptive integration. These concepts can be applied, for example, to control the reverberation transmit gain of artificial reverberation used in acoustic reproduction in a car (reverberation transmit gain: e.g., (artificial) reverberation (transmit) gain).

[0034] Reverberation is a highly complex element in acoustic and audio signal processing, and embodiments focus on its perceived intensity and control, for example, by adjusting the reverberation transmission gain. Some embodiments may be implemented independently of other aspects, such as the frequency-dependent reverberation time of the impulse response or its correlation across channels. Other embodiments may take these aspects into consideration, for example.

[0035] A schematic diagram of a specific embodiment is shown in Figure 2. The first step may include, for example, reverberation removal and audio feature extraction. In particular, Figure 2 shows a block diagram of a reverberation control algorithm according to one embodiment. Feature values ​​are input to a model for predicting the perceived reverberation intensity in the input signal. The model uses linear regression and is trained on listening test data. The model output is post-processed with signal adaptive scaling coefficients that are manually adjusted for the characteristics of the newly added artificial reverberation to address the altered interaction between the excitation signal and the reverberation due to the different characteristics of the artificial reverberation and the "original" reverberation. It should be noted that data-driven modeling of perceived reverberation intensity for artificial reverberation (as is done for original reverberation) requires listening test data for multiple parameter settings and cannot be adapted when these settings are changed. Therefore, such an approach is undesirable.

[0036] The equivalent reverberation level is post-processed with a real-time signal-adaptive time integral. The result is applied to calculate the reverberation transmit gain using a function fitted to a preferred transmit gain tuned by a professional listener. The calculated reverberation transmit gain controls the level of artificial reverberation applied to the dereverberated input signal. Dereverberation is applied to attenuate the reverberation signal component of the input because adding artificial reverberation to an input signal with a large amount of "original" reverberation would result in a loss of clarity, an excessive perception of reverberation, or an aesthetically undesirable superposition of the two reverberation signals.

[0037] The results of the listening test are shown below. The listening test presented 27 volume-normalized audio signals with durations ranging from 4.8 seconds to 14.7 seconds. The items were selected so that only small changes in spatial cues (amount and energy distribution of reverberation in a stereo panorama) would be apparent. The set of items ranged from very dry to very reverberant and included recordings of various musical genres, as well as recordings of single instruments and dry speech.

[0038] Participants were asked to rate perceived reverberation intensity and ensemble width by adjusting sliders on a separate unipolar scale ranging from 1 to 9. The labels “very low,” “low,” “medium,” “high,” and “very high” were evenly distributed on the scale. Ensemble width, an additional attribute, was provided to facilitate listeners in judging reverberation intensity independently of stereo panorama. The test began with a training session in which three stimuli were presented to give examples, for example, of low reverberation and wide ensemble width, and vice versa. Fifteen listeners participated in the test.

[0039] Figure 3 shows the average evaluation and overall mean per listening test item with a 95% confidence interval according to one embodiment. In particular, Figure 3 shows the average value and 95% confidence interval for reverberation evaluation. The confidence interval is: TIFF0007872853000012.tif1746(1) This was obtained according to the following. Here, TIFF0007872853000013.tif64m is the estimated average. It is TIFF0007872853000014.tif619, TIFF0007872853000015.tif58 is from the t direction Displacement values ​​with degrees of freedom for TIFF0007872853000016.tif411 (here, TIFF0007872853000017.tif33=0.05), TIFF0007872853000018.tif54 is the estimated baseline deviation, TIFF0007872853000019.tif33 is a sample size [8].

[0040] The following shows a linear regression model according to one embodiment. The following describes a computational model and its training for predicting the perceived reverberation intensity of an audio signal according to a specific embodiment.

[0041] First, we provide a model description based on one embodiment. The linear regression model is used to determine perceived reverberation intensity. TIFF0007872853000020.tif53, input features A linear combination of TIFF0007872853000021.tif56 TIFF0007872853000022.tif2152(2) This is applied to estimate as follows: where the model coefficients are TIFF0007872853000023.tif66, bias value TIFF0007872853000024.tif66, the number of input features The filename is TIFF0007872853000025.tif44.

[0042] Next, we will describe the audio feature extraction according to the embodiment. The inputs to the linear regression model are the inter-channel coherence (ICC) and spectral flatness measure (SFM) of the subbands, calculated from the short-term Fourier transform (STFT) coefficients. As shown in Table 1, the ICC is calculated at 5, and the SFM is calculated for four frequency bands.

[0043] Table 1 shows the frequency band separation of ICC and SFM according to one embodiment.

[0044] [Table 1]

[0045] The STFT coefficient is calculated from an audio signal sampled at 48 kHz with a frame size of 1024 samples, a hop length of 512 samples, and no zero padding. STFT frames with a volume below -65 LUFS (Units of Volume relative to Full Scale) were removed to make the model training independent of silent periods, e.g., speech pauses. ICC[9] TIFF0007872853000027.tif21101(3) It is calculated from both channel signals of a 2-channel stereo input signal according to the formula. TIFF0007872853000028.tif724 and TIFF0007872853000029.tif724 represents the automatic power spectral density (PSD) of the left and right channels, respectively. TIFF0007872853000030.tif724 is a time index In TIFF0007872853000031.tif34 TIFF0007872853000032.tif5 is a crossed PSD where all PSDs are accumulated across the frequency bins corresponding to the 4th frequency band. For example, TIFF0007872853000033.tif618 is the first audio channel of the audio input signal. TIFF0007872853000034.tif54th frequency band TIFF0007872853000035.tif42 shows the complex spectral bin, TIFF0007872853000036.tif618 is the second audio channel of the audio input signal. TIFF0007872853000037.tif54th frequency band If we are showing the 42nd complex spectral bin of TIFF0007872853000038.tif, TIFF0007872853000039.tif724, TIFF0007872853000040.tif724, and TIFF0007872853000041.tif724 is, for example, It is defined as follows: TIFF0007872853000042.tif1572TIFF0007872853000043.tif1575TIFF0007872853000044.tif1572. Here, the time index is As TIFF0007872853000045.tif34, TIFF0007872853000046.tif521 is Show the complex conjugate of TIFF0007872853000047.tif520. TIFF0007872853000048.tif521 is The complex conjugate of TIFF0007872853000049.tif519 is shown. TIFF0007872853000050.tif44 is, for example, a frequency band Frequency bins in TIFF0007872853000051.tif43 The number of downmix squared STFT amplitude coefficients for TIFF0007872853000052.tif55 can be shown.

[0046] For example, in certain embodiments, the frequency band may include, for example, one or more frequency bins for each audio channel (for example, in certain embodiments, the frequency band may simply be, for example, a single bin). For example, in certain embodiments, equation (3) may be used, for example, to obtain the inter-channel correlation of the aforementioned frequency band.

[0047] SFM (see

[10] ) is, TIFF0007872853000053.tif26104(4) It is calculated as the ratio of the geometric mean to the arithmetic mean, TIFF0007872853000054.tif13100(5) That is the case. Here, TIFF0007872853000055.tif610 is a compression function, for example logarithmic, or square root, or It is TIFF0007872853000056.tif723, TIFF0007872853000057.tif929 is a time index The file TIFF0007872853000058.tif34 is available. TIFF0007872853000059.tif44 bin-based frequency band In TIFF0007872853000060.tif43 TIFF0007872853000061.tif42nd frequency bin These are the downmix squared STFT amplitude coefficients of the left and right input channels of TIFF0007872853000062.tif68. (Therefore, TIFF0007872853000063.tif44 is, for example, a frequency band Frequency bins of TIFF0007872853000064.tif43 The number of TIFF0007872853000065.tif68 can be shown.) For example, in an embodiment having two audio channels, the downmix squared STFT amplitude coefficient TIFF0007872853000066.tif929 is, for example, a downmix STFT amplitude coefficient. The result may also be the square of TIFF0007872853000067.tif827, and the downmix STFT amplitude coefficient. TIFF0007872853000068.tif827 is, for example, TIFF0007872853000069.tif5 may also be the result of combining (e.g., averaging) the amplitude coefficients of two audio channels in the frequency bin of the 4th frequency band.

[0048] In this embodiment, features were extracted for each frame (e.g., for each block), and then the arithmetic mean and standard deviation were calculated to obtain a single value for each block, which was used to train the regression model.

[0049] For example, each block may include audio data for playing back one second of a recording. A block may include spectral data for 94 time points (for example, with respect to 94 frames, for example, assuming in a particular embodiment that each frame has a duration of 21.3 milliseconds, and the frames have a 50% overlap, this would be approximately 94 frames / second). In other embodiments, a block may include spectral data for any other number of time points. In one example, the spectrum may be divided into, for example, five frequency bands and / or four frequency bands, as outlined in Table 1.

[0050] For example, taking ICC into consideration, the ICC value is determined for each of the five frequency bands in Table 1, and for each of the 94 time points. For example, for each of the five frequency bands, the arithmetic mean of the 94 ICCs in the block is determined, and thus, five arithmetic mean ICC values ​​for the five frequency bands may be obtained. And / or, for example, the standard deviation of the 94 ICCs in the block is determined for each of the five frequency bands, and thus, five standard deviation ICC values ​​for the five frequency bands may be obtained. This results in, for example, 10 ICC-related feature values ​​for the five frequency bands of the block.

[0051] For example, taking SFM into consideration, SFM values ​​are determined for each of the four frequency bands in Table 1, and for each of the 94 time points. For example, for each of the four frequency bands, the arithmetic mean of the 94 SFMs in the block is determined, and thus, the four arithmetic mean SFM values ​​for the four frequency bands may be obtained. And / or, for example, the standard deviation of the 94 ICCs in the block is determined for each of the four frequency bands, and thus, the four standard deviation SFM values ​​for the four frequency bands may be obtained. This results in, for example, eight SFM-related feature values ​​for the four frequency bands of the block.

[0052] For a combination of both ICC and SFM examples, for example, 18 feature values ​​for a block can be obtained.

[0053] Next, we will describe the model training according to the embodiment. Processing short blocks of data is essential for real-time processing, especially for transient signals, and for increasing the amount of available training data. Each item was windowed with a training block length of 6 seconds with zero overlap. Items shorter than 12 seconds were treated as a single block of their entire length. Performance when training with overlapping blocks was also evaluated, and no improvement in model accuracy was observed.

[0054] The training dataset was augmented by adding a second excerpt with similar spatial characteristics to several songs used in listening tests. Furthermore, several additional key items that were poorly predicted were added, along with reference ratings provided by one expert listener. To ensure the model predicts small values ​​for dry and clean speech signals, recordings of male and female speakers were added to the dataset with reference annotations set to -1 MOS (Mean Opinion Score). By augmenting the dataset and using multiple blocks for each item, a total of 100 observations were obtained.

[0055] To take into account the nonlinear relationship between listening test data and feature values, indexing was applied to the mean and standard deviation of the feature values. The index was determined by manual adjustment based on the evaluation of the residual plot, which reveals nonlinearities that cannot be modeled by linear regression by definition.

[0056] Model training is predictive. The evaluation was performed using a standard least squares (OLS) algorithm to minimize the mean squared error between TIFF0007872853000070.tif53 and the mean score obtained from the listening test. The evaluation was performed using cross-validation with a leave-one-out method

[11] , where the model was trained on all but one observation and evaluated on the excluded item. This procedure was repeated for each item.

[0057] Next, we will describe the feature selection according to the embodiment. Training began with all available values, such as the arithmetic mean and standard deviation for each subband feature. The subband features may be, for example, ICC or SFM for the aforementioned subbands (frequency bands).

[0058] To avoid overfitting and reduce computational load, we eliminated unhelpful independent variables one by one by evaluating their p-values. Low p-values ​​indicate that there is most likely no relationship between the corresponding feature value and the listening test results (see

[11] ).

[0059] For example, in the above example with 18 feature values ​​(10 ICC-related features and 8 SFM-related features), gradually removing all irrelevant features may result in a model with, for example, 4 input variables (for example, 14 other irrelevant or less relevant input variables / features can be gradually removed, for example, by performing multivariate regression analysis (see, for example,

[11] ). The performance of this model is shown in Figure 4.

[0060] In particular, Figure 4 shows the predicted reverberation intensity against the observed reverberation intensity in one embodiment. Predictions for the training blocks of listening test items are shown as dots, and predictions obtained for the training blocks of the extended dataset (items from the extended dataset) are represented by triangles. The average prediction for each item is shown as a cross. Markers representing blocks of the same item have the same listening test evaluation. The plot reveals a good correlation between predictions and observations, with a correlation coefficient of 0.75 and an average error of 1.27 MOS. For listening test evaluations around 3 MOS, high evaluations are estimated to be slightly low, while there is a tendency towards values ​​that are too high.

[0061] The following provides a reverberation transmission gain calculation according to one embodiment. In particular, it describes the calculation of a preferred reverberation transmission gain for artificial reverberation given the perceived intensity of the primary reverberation in the input signal. Because the original reverberation (of the input signal) and the secondary (artificial) reverberation do not coincide, equal reverberation levels can result in different perceived reverberation intensities. Therefore, considering the perceived reverberation intensity, the reverberation transmission gain cannot be calculated directly.

[0062] Data-driven modeling of the perceived intensity of artificial reverberation is not feasible because it requires subjective data for different reverberation settings.

[0063] In the embodiment, the scaling factor is manually adjusted. TIFF0007872853000071.tif55 and TIFF0007872853000072.tif55 is introduced, for example, to compensate for signal-dependent effects, as described below. For the purpose of explaining the process, the equivalent reverberation level The intermediate quantity TIFF0007872853000073.tif64 is defined as follows: TIFF0007872853000074.tif1148(6)

[0064] The equivalent reverberation level represents the desired level of artificial reverberation for a given input signal that yields similar reverberation intensity to both artificial and original reverberation. The equivalent reverberation level is then converted to reverberation transmit gain using a mapping function determined in a separate tuning experiment.

[0065] Next, we will describe the equivalent reverberation level according to the embodiment. Signals with strong transients, such as drum sounds, produce a higher perceived reverberation intensity than steady-state signals and are therefore less effective at masking reverberation.

[0066] Artificial reverberations with large time constants may require lower reverberation levels to evoke a similar intensity perception to that of the original reverberation. Several embodiments provide novel spectral transient measurements (STMs) for quantifying the intensity of transients within a signal as follows: TIFF0007872853000075.tif14132(7) During the ceremony, TIFF0007872853000076.tif2182(8) That is the case. TIFF0007872853000077.tif611 demonstrates time-based recursive averaging and performs smoothing of subband signals.

[0067] For example, in a particular embodiment, the frequency bands may be, for example, the frequency bands in the left column or the right column of Table 1. For example, alternatively, other frequency band / frequency band configurations may be used. Each of the frequency bands is, for example, then a time index All of the aforementioned frequency bands for TIFF0007872853000078.tif34 can be included, for example, the time-frequency bin.

[0068] The STM single value is obtained by averaging the STM values ​​in the 23–70Hz and 5.2–21kHz frequency bands. This band selection focuses on transients caused by percussion instruments, while transients from other instruments, such as the piano, are not given much consideration.

[0069] Scaling factor TIFF0007872853000079.tif55 is calculated from STM as follows: TIFF0007872853000080.tif1192(9) Here, TIFF0007872853000081.tif512 shows the average STM values ​​of a 1-second block subsequently cut off according to the following: TIFF0007872853000082.tif1261(10) TIFF0007872853000083.tif46, TIFF0007872853000084.tif44, TIFF0007872853000085.tif411, and TIFF0007872853000086.tif411 is an adjustable parameter that is tuned according to the characteristics of the artificial reverberation.

[0070] If the reverberation time of the "original" reverberation exceeds that of one of the artificial reverberations, a larger equivalent reverberation level is required. Second scaling factor TIFF0007872853000087.tif55 is designed to adjust the estimated reverberation intensity of input signals containing a large amount of reverberation with long reverberation times, such as classical music. Such items are assumed to be largely uncorrelated due to the large amount of diffuse sound components. TIFF0007872853000088.tif55 is calculated as follows: TIFF0007872853000089.tif1158(11) Here, TIFF0007872853000090.tif510 is the average ICC of all frequency bands and frames within a 1-second block, which will then be truncated as follows: TIFF0007872853000091.tif1057(12) During the ceremony, TIFF0007872853000092.tif46, TIFF0007872853000093.tif45, TIFF0007872853000094.tif411, and TIFF0007872853000095.tif412 is an adjustable parameter.

[0071] The following describes signal-adaptive time integration according to the embodiment. To control the reverberation transmission gain in real time, it is desirable to respond to changes in the input signal without introducing significant modulation of the reverberation level. To address this with low latency and low computational load, predictions are computed from 8-second blocks, each with a 7-second overlap, so that new predictions are computed at a rate of 1 Hz. Low input levels, e.g., signals below -65 LUFS, are not fed to the model as they were during training. Equivalent reverberation level TIFF0007872853000096.tif613(6) is temporally smoothed using recursive averaging with a unipolar IIR filter to obtain the following: TIFF0007872853000097.tif10108(13) Here, the smoothing coefficient is Let's call it TIFF0007872853000098.tif34.

[0072] To avoid excessively high reverberation levels in transitions between input signals with very different characteristics, track changes are detected and fast adaptation of the equivalent reverberation level is performed. Fast adaptation reduces the model block length to 1 second and smooths the coefficient. This is done by increasing TIFF0007872853000099.tif34. For detection, a change in the variance of the STM is detected, because a change in the amount of transients is a good indicator for, for example, a transition from music to speech, which requires a much lower reverberation level.

[0073] Figure 5 illustrates the real-time processing of an input file for explanation, consisting of a 25-second classical music passage and a rapid transition to pop music, as can be observed in the upper plot showing the spectrogram according to the embodiment. In particular, the upper part of Figure 5 shows the spectrogram of the audio signal that transitioned from classical music to pop music after 25 seconds. The middle of Figure 5 shows the equivalent reverberation level before (solid line) and after (dashed line) temporal smoothing. The lower part of Figure 5 shows the smoothing coefficient (solid line) and the total scaling value (dashed line).

[0074] The central subplot shows the equivalent reverberation levels before (solid line) and after (dashed line) temporal smoothing, with values ​​of approximately 7 MOS for classical music and 2.2 MOS for pop music. The lower subplot shows the total scaling value. The product of the scaling values, called TIFF0007872853000100.tif619, is shown as a dashed line. Due to the low correlation of the classical music items, the total scaling value is greater than 1 until the track changes. The increase in the amount of transient at t=25 seconds triggers track change detection, reducing the total scaling value to approximately 0.7.

[0075] The smoothing coefficients are plotted as solid lines in the subplot below. TIFF0007872853000101.tif34 decreases after an 11-second adjustment phase because the volume and equivalent reverberation level are rather static. Track change detection at t=25 seconds is Temporarily increase TIFF0007872853000102.tif34 to enable rapid adaptation of equivalent reverberation levels.

[0076] Next, we will describe the conversion from reverberation prediction to reverberation transmission gain according to the embodiment. For example, as a final step, the transmit gain of the reverberation processor may be calculated taking into account, for instance, the equivalent reverberation level. For this purpose, a second listening test was conducted, in which five expert listeners adjusted the transmit gain of two artificial reverberations simulating the acoustic environments of a concert hall (T60 = 2.2 seconds) and a jazz club (T60 = 1.3 seconds) inside a car, according to their preferences.

[0077] Figure 6 shows the reference reverberation transmit gain over equivalent reverberation levels with a fitted polynomial according to one embodiment. In particular, Figure 6 shows the average transmit gain obtained over equivalent reverberation levels with a 95% confidence interval according to equation (1) for all 30 test items of the two artificial reverberations. Since longer reverberation tails result in higher perceived reverberation intensity with transients requiring less masking and lower reverberation transmit gain, the adjustable parameters of equations (9) and (11) were adjusted so that strong transients and lower correlations result in lower equivalent reverberation levels for the concert hall compared to the jazz club. As a result, one single mapping curve can be used for both reverberations. Using a value of 0.81, Spearman's rank correlation coefficient confirms a monotonic relationship between the two variables that can be approximated by a cubic polynomial with low error.

[0078] Using gain adjustment, polynomial mapping is performed as shown in Figure 6. TIFF0007872853000103.tif652 was fitted. This smoothed the equivalent reverberation level. Predicted reverberation transmit gain for TIFF0007872853000104.tif66 Convert to TIFF0007872853000105.tif43.

[0079] Below, we will discuss the evaluation methods. To evaluate the polynomial mapping, a leave-one-out cross-validation technique was used, meaning that all but one pair of values ​​were used to fit the polynomial function, with the last data pair playing a role in obtaining the error for this particular data point. This process was repeated for all listening test items with a mean absolute error (MAE) of 1.74 dB. According to [7], such deviations from the favorable reverberation transmit gain result in a negligible decrease in aesthetic quality. The limits of the 95% confidence intervals obtained according to equation (1) are 1.27 dB and 2.21 dB. The absolute maximum error corresponds to 5.11 dB, and the correlation between the predicted and observed values ​​is 0.89.

[0080] Figure 7 shows the equivalent reverberation level with respect to reverberation transmission gain error according to one embodiment. The scatter plot shown in Figure 7 shows the reverberation transmission gain error over the equivalent reverberation level and exhibits a uniform distribution of errors that exceeds the full range with only one outlier in 2.6MOS.

[0081] Concepts for controlling the level of artificial reverberation are provided. Some of the provided concepts employ models to predict perceived reverberation intensity based on linear regression models trained on listening test data. This method uses manually tuned adaptations to various parameter settings of the artificial reverberation. Fast adaptation to changes in input signal characteristics is performed by signal adaptation time integration. This algorithm was evaluated in in-car acoustics reproduction applications. It can also predict a favorable reverberation transmission gain of 1.74 dB MAE.

[0082] Next, we will describe an extended form of the embodiment provided above. For example, other application scenarios can be implemented.

[0083] The embodiments may be applied, for example, to control reverberation in various other application scenarios. In binaural sound reproduction, artificial reverberation is applied, for example, to support the sense of externalization, in which the listener localizes acoustic events, objects, and sound sources outside of their head. This is in contrast to standard sound reproduction using headphones, where sound is typically perceived as coming from inside the head.

[0084] Artificial reverberation is used in music production, applied to the mixing of multiple sound sources and individual sound sources. The proposed method can also be extended to predict the amount of perceived reverberation for individual sound sources or individual components of a mixed signal (e.g., groups of sound sources). Information on the amount for each individual component enables automated control or artificial reverberation in sound production (music production, film audio, podcasts, and other user-generated content).

[0085] Other embodiments may implement alternative models, for example. Linear models are, for example, other data-driven models, for example, arbitrary functions It may be replaced with TIFF0007872853000106.tif934. Here, the scalar output TIFF0007872853000107.tif64 is an input vector TIFF0007872853000108.tif34, and, TIFF0007872853000109.tif64 and target output Model parameters determined by optimizing the criteria between TIFF0007872853000110.tif54 (e.g., minimizing the mean squared error) Given TIFF0007872853000111.tif65, it is calculated. Another well-known class of such trainable models is artificial neural networks, in particular deep neural networks (DNNs). Parameters The larger the set of TIFF0007872853000112.tif65, the more data is required to train the model, and as a result, the performance is not only better for the training data, but also for new data that was not used during training. It should be noted that this is also good for TIFF0007872853000113.tif613. This property is called generalization. However, with more training data available, other models such as DNNs may yield better predictions (with higher accuracy).

[0086] Such DNNs are implemented by combining layers of processing units, each unit having trainable parameters. Commonly used types of layers are convolutional layers, high-density layers (also called fully connected layers), and iterative layers. They differ in the types of units implemented and how these units are connected to one another. Additional types of layers are used to support the training process, where training refers to numerically optimizing parameters. This refers to the process of optimizing TIFF0007872853000114.tif65.

[0087] Further embodiments are provided below. An apparatus and / or method is provided for controlling the level of an artificially generated reverberation signal added to an audio signal, using a model of the perceived intensity of reverberation in the original input signal.

[0088] According to one embodiment, the input signal may be de-reverbed, for example, to attenuate the original reverberation signal component.

[0089] In one embodiment, a linear regression model may be employed, for example, with different audio feature values ​​as input.

[0090] According to one embodiment, subband SFM and / or subband ICC may be used, for example, as input audio features.

[0091] In one embodiment, adjustable scaling values ​​can be employed to modify the model prediction, which depend on audio feature values ​​to compensate for the modified interaction of the artificial reverberation and the de-reverberation input signal compared to the original reverberation and direct signal.

[0092] According to one embodiment, for example, adjustable scaling values ​​that depend on subband ICC and subband STM can be employed.

[0093] In one embodiment, a ring buffer with high overlap for real-time processing can be employed, for example.

[0094] According to one embodiment, adaptive temporal smoothing can be employed, for example, to adjust the amount of smoothing in accordance with changes in the input signal.

[0095] In one embodiment, changes in volume and model predictions may be evaluated, for example, to control the smoothing coefficient of the temporal recursive averaging.

[0096] According to one embodiment, track changes may be detected in response to changes in the variance of audio features, for example, to temporarily reduce the smoothing coefficient of temporal recursive averaging and reset the ring buffer.

[0097] In one embodiment, a mapping function adapted to listening test data can be used, for example, to convert a tuned model prediction into reverberation transmission gain.

[0098] The proposed concept can be applied, for example, to acoustic reproduction in automobiles to emulate acoustic environments with larger size and pleasing spatial characteristics. Emulation of the acoustic environment is achieved by processing the audio input signal so that the reverberation signal component of the output signal is perceptually similar to the reproduction of the direct signal component in the new environment. This implementation has low latency and low workload.

[0099] While some embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, embodiments described in the context of method steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such devices.

[0100] Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. Implementation may be carried out using a digital storage medium containing electronically readable control signals, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, which may (or may) cooperate with a programmable computer system to perform the respective method. Therefore, the digital storage medium may be computer-readable.

[0101] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein can be performed.

[0102] Generally, embodiments of the present invention can be implemented as a computer program product having program code, the program code operates to perform one of the methods when the computer program product is executed on a computer. The program code can be stored, for example, in a machine-readable carrier.

[0103] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0104] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0105] Accordingly, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) containing a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-temporary.

[0106] Therefore, a further embodiment of the method of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0107] Further embodiments include processing means configured or adapted to perform one of the methods described herein, such as a computer or a programmable logic device.

[0108] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0109] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0110] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the method herein. In some embodiments, a field-programmable gate array can cooperate with a microprocessor to perform one of the methods herein. Generally, the method is preferably performed by any hardware device.

[0111] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0112] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0113] The embodiments described above are merely illustrative of the principles of the present disclosure. Modifications and variations of the configurations and details described herein will be obvious to those skilled in the art. Therefore, it is intended to be limited only by the pending claims and not by any specific details presented as part of the description and explanation of the embodiments herein.

[0114] References [1] Paulus, J., Uhle, C., and Herre, J., “Perceived Level of Late Reverberation in Speech and Music,” in Proc. AES 130th Conv., 2011. [2] Gardner, W. G. and Griesinger, D., “Reverberation level matching experiments,” in Proc. Sabine Centennial Symposium, Acoust. Soc. of Am., 1994. [3] Leonard, B., King, R., and Sikora, G., “The Effect of Playback System on Reverberation Level Preference,” in Audio Engineering Society Convention 134, 2013. [4] Leonard, B., King, R., and Sikora, G., “The Effect of Acoustic Environment on Reverberation Level Preference,” in Audio Engineering Society Convention 133, 2012. [5] Sarroff, A. M. and Bello, J. P., “Toward a Computational Model of Perceived Spaciousness in Recorded Music,” J. Audio Eng. Soc, 59(7 / 8), pp. 498-513, 2011. [6] Uhle, C., Paulus, J., and Herre, J., “Predicting the Perceived Level of Late Reverberation Using Computational Models of Loudness,” in Proc. 17th Int. Conf. on Digital Signal Process. (DSP), 2011. [7] Paulus, J., Uhle, C., Herre, J., and Hoepfel, M., “A Study on the Preferred Level of Late Reverberation in Speech and Music,” in Proc. of the 60th Int. Conf. of Audio Eng. Soc., 2016. [8] Gilleland, E., Confidence Intervals for Forecast Verification, National Center For Atmospheric Research, Boulder, Colorado, 2010. [9] Allen, J., Berkeley, D., and Blauert, J., “Multimicrophone Signal-Processing Technique to Remove Room Reverberation from Speech Signals,” J. Acoust. Soc. Am., 62, 1977.

[10] Gray, A. H. and Markel, J. D., “A Spectral-Flatness Measure for Studying the Autocorrelation Method of Linear Prediction of Speech Analysis,” IEEE Trans. Acoust., Speech, and Sig. Process., 22, pp. 207-217, 1974.

[11] James, G., Witten, D., Hastie, T., and Tibshirani R., editors, “An Introduction to Statistical Learning: with Applications in R,” number 103 in Springer Texts in Statistics, Springer, New York, 2013.

Claims

1. A device for processing an audio input signal including one or more audio channels in order to acquire an audio output signal, wherein the device is A reverberation gain determination device (110) configured to determine reverberation gain information in accordance with the aforementioned audio input signal, The system includes a signal processor (120) configured to acquire the audio output signal according to the reverberation gain information by adding artificial reverberation to the audio input signal or a preprocessed audio signal dependent on the audio input signal, The reverberation gain determination device (110) is configured to determine the reverberation gain information in accordance with the estimation of the perceived intensity of reverberation in the audio input signal. The apparatus is configured such that when the reverberation gain determination device (110) receives information about one or more features of the audio input signal, it determines the reverberation gain information by employing a model that returns the estimation of the perceived intensity of the reverberation in the audio input signal.

2. The apparatus according to claim 1, wherein the model adopted by the reverberation gain determination device (110) is a linear regression model, which uses one or more feature values ​​of the one or more features of the audio input signal as input for the linear regression model.

3. The apparatus according to claim 1, wherein the reverberation gain determination device (110) is configured to determine the reverberation transmission gain as reverberation gain information by mapping the estimation of the perceived intensity of the reverberation in the audio input signal according to a mapping function.

4. The apparatus according to claim 1, wherein one or more of the features of the audio input signal depend on the inter-channel correlation of at least one of the subbands of two audio channels of the one or more audio channels of the audio input signal.

5. The apparatus according to claim 1, wherein one or more of the features of the audio input signal depend on a spectral flatness measure of at least one of the subbands of the one or more audio channels of the audio input signal.

6. The reverberation gain determination device is configured to determine the estimated perceived intensity of the reverberation in the audio input signal by using the model, The reverberation gain determination device is configured to determine one or more scaling coefficients according to one or more of the characteristics of the audio input signal. The apparatus according to claim 1, wherein the reverberation gain determination device is configured to determine the reverberation gain information in accordance with the estimation of the perceived reverberation intensity and in accordance with one or more scaling coefficients.

7. The one or more scaling factors mentioned above are The inter-channel correlation of at least one of the subbands of one or more audio channels of the one or more audio channels of the audio input signal, The presence of transient signal components in at least one of the one or more audio channels of the audio input signal, The apparatus according to claim 6, which depends on at least one of the following.

8. The signal processor (120) is configured to generate the pre-processed audio signal by removing reverberation from the audio input signal in order to attenuate the original reverberation signal component of the audio input signal. The apparatus according to claim 1, wherein the signal processor (120) is configured to acquire the audio output signal in accordance with the reverberation gain information by adding the artificial reverberation to the preprocessed audio signal.

9. The signal processor (120) is configured to generate the preprocessed audio signal by performing temporal smoothing of the audio input signal. The apparatus according to claim 1, wherein the signal processor (120) is configured to acquire the audio output signal in accordance with the reverberation gain information by adding the artificial reverberation to the preprocessed audio signal.

10. The apparatus according to claim 9, wherein the signal processor (120) is configured to adjust the amount of temporal smoothing in accordance with the change in the audio input signal.

11. The signal processor (120) The change in volume of the aforementioned audio input signal, The change in the distribution of one or more features of the audio input signal, The apparatus according to claim 10, configured to adjust the amount of the temporal smoothing according to at least one of the following.

12. The device comprises a ring buffer for receiving the audio input signal or the pre-processed audio signal, The apparatus according to claim 1, wherein the signal processor (120) is configured to process the audio input signal or the pre-processed audio signal in the ring buffer to obtain the audio output signal.

13. A method for processing an audio input signal in order to obtain an audio output signal, wherein the method is The reverberation gain information is determined according to the aforementioned audio input signal, By adding artificial reverberation to the aforementioned audio input signal or a pre-processed audio signal dependent on the aforementioned audio input signal, the audio output signal is obtained according to the reverberation gain information. The determination of the reverberation gain information is performed according to the estimation of the perceived intensity of reverberation in the audio input signal. A method for determining the reverberation gain information, wherein, upon receiving information about one or more features of the audio input signal, the model returns the estimated intensity of the reverberation in the audio input signal.

14. A computer program for carrying out the method described in claim 13, when executed on a computer or signal processor.

Images