Apparatus and method for automatically controlling reverberation level using perceptual model

By using a reverb gain determiner and signal processor in audio signal processing, the reverb transmission gain is adjusted in real time, and the problem of difficulty in controlling reverb level in the prior art is solved, improving the aesthetic quality of the audio signal and clear sound reproduction Spend.

CN119096563APending Publication Date: 2024-12-06FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028418.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-16
Filing Date
2023-03-14
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing audio signal processing technologies are difficult to control reverberation levels in real time and effectively, especially in music recordings, resulting in an impact on aesthetic quality.

Method used

Using a device and method, the reverb gain information is determined based on the audio input signal through a reverb gain determiner, and the signal processor is used to add artificial reverb on the computer or signal processor to adjust the reverb sending gain to match the preferred level , realize real-time reverb control.

Benefits of technology

Real-time automatic control of the reverb level is realized, the aesthetic quality of the audio signal is improved, the reverb characteristics that are consistent with the preferred level are ensured, the superposition of the reverb signal is reduced, and the clarity of sound reproduction is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096563A_ABST
    Figure CN119096563A_ABST
Patent Text Reader

Abstract

An apparatus for processing an audio input signal comprising one or more audio channels to obtain an audio output signal according to an embodiment is provided. The apparatus comprises a reverberation gain determiner (110) configured to determine reverberation gain information from an audio input signal. Furthermore, the apparatus comprises a signal processor (120) configured to obtain the audio output signal by adding an artificial reverberation to the audio input signal or adding an artificial reverberation to the preprocessed audio signal in accordance with the audio input signal in accordance with the reverberation gain information.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] manual

[0002] The present invention relates to audio signal processing, and in particular to apparatus and methods for automatically controlling reverberation levels, for example by using perceptual models.

[0003] Reverberation is a very complex element in acoustic and audio signal processing, and its perceived intensity and its control are of particular interest. Audio signals like music recordings or radio broadcasts can be processed, for example, with the aid of artificial reverberation to simulate the acoustic properties of a specific environment (e.g., a concert venue or hall). The input signal can be, for example, a mixture of a direct signal and an ambient signal. The direct signal can be, for example, a recording of singing, musical instruments, and sound events (such as gunshots and alarm sirens). The term "direct sound" indicates that these sounds are directional and can be localized as coming from one direction. Ambient (or diffuse) sound components can be, for example, reverberation and ambient sounds that are not perceived as coming from a specific direction, such as wind noise and rain. In music recordings, reverberation is the most important source of ambient sound.

[0004] The perceived reverberation level depends on the input signal and the reverberation impulse response, e.g., the length of the pre-delay and reverberation tail (see [1]). Non-stationary input signals with fast attacks of transients and subband envelopes (e.g., drums) produce higher reverberation intensity.

[0005] Input signals with fast decaying envelopes and silent sections are less effective at masking reverberation (see [2]). When reverberation time and pre-delay are small, the input signal overlaps with the reverberation signal and partially masks the reverberation signal to a greater extent than when reverberation time and pre-delay are large. In addition, the acoustic environment (see [3]), the playback system (see [4]), and other aesthetic aspects related to the musical genre influence the preferred gain parameter setting.

[0006] A prediction model that uses listening test data to tune multiple spatial characteristics has been described in [5], but the proposed model is not applied in real time and the model itself is more complex since it uses up to 12 audio features, but its performance is comparable to the presented algorithm using only 3 features. The model of the perceptual level of late reverberation in (see [6]) uses the partial loudness of the direct and reverberant components of the input audio signal. The study of the perceptual level of artificial late reverberation in [1] showed that the perceptual level depends on the reverberation time and the input signal, rather than the inter-channel correlation of the impulse response, and that the differences in ratings between individuals and repeated tests of the same individual are similar. The preferred level was studied in [7], where it was concluded that the aesthetic quality is affected more when the applied reverberation is above the preferred level than when it is below the preferred level.

[0007] The object of the invention is to provide an improved concept for automatically controlling the reverberation level. The object of the invention is achieved by a device according to claim 1 , a method according to claim 19 and a computer program according to claim 20 .

[0008] According to an embodiment, a device for processing an audio input signal including one or more audio channels to obtain an audio output signal is provided. The device includes a reverberation gain determiner configured to determine reverberation gain information according to the audio input signal. In addition, the device includes a signal processor configured to obtain the audio output signal by adding artificial reverberation to the audio input signal or to a preprocessed audio signal according to the audio input signal according to the reverberation gain information.

[0009] Furthermore, according to an embodiment, a method for processing an audio input signal to obtain an audio output signal is provided. The method comprises:

[0010] -determining reverberation gain information based on the audio input signal. And:

[0011] - obtaining an audio output signal by adding artificial reverberation to the audio input signal or to a preprocessed audio signal according to the audio input signal, according to the reverberation gain information.

[0012] Furthermore, a computer program is provided for implementing the above method when executed on a computer or a signal processor.

[0013] In the following, embodiments of the present invention are described in more detail with reference to the accompanying drawings, in which:

[0014] Figure 1 A device for processing an audio input signal comprising one or more audio channels to obtain an audio output signal according to an embodiment is shown.

[0015] Figure 2 A block diagram of a reverberation control algorithm according to an embodiment is shown.

[0016] Figure 3 Shown are the mean ratings for each listening test item and the overall mean with 95% confidence intervals according to an embodiment.

[0017] Figure 4 Predicted reverberation intensity compared to observed reverberation intensity according to an embodiment is shown.

[0018] Figure 5 Shows Figure 5 The top is the spectrogram of the audio signal transitioning from classical music to pop music after 25 seconds; Figure 5 The middle is the equivalent reverberation level before and after time smoothing; and Figure 5At the bottom are the smoothing coefficient and the total scaling value according to an embodiment.

[0019] Figure 6 Reference reverberation send gain over equivalent reverberation level with a fitting polynomial according to an embodiment is shown.

[0020] Figure 7 The relationship between the equivalent reverberation level and the reverberation send gain error according to an embodiment is shown.

[0021] Figure 1 A device for processing an audio input signal comprising one or more audio channels to obtain an audio output signal according to an embodiment is shown.

[0022] The apparatus comprises a reverberation gain determiner 110 configured to determine reverberation gain information from an audio input signal.

[0023] Furthermore, the apparatus comprises a signal processor 120 configured to obtain an audio output signal by adding artificial reverberation to the audio input signal or to a preprocessed audio signal according to the audio input signal according to the reverberation gain information.

[0024] According to an embodiment, the reverberation gain determiner 110 may, for example, be configured to determine the reverberation gain information based on an estimate of the perceived strength of the reverberation in the audio input signal.

[0025] In an embodiment, the reverberation gain determiner 110 may, for example, be configured to determine the reverberation gain information by employing a model that returns an estimate of the perceived strength of the reverberation in the audio input signal upon receiving information about one or more characteristics of the audio input signal.

[0026] According to an embodiment, the model adopted by the reverberation gain determiner 110 may be, for example, a linear regression model, using one or more feature values ​​of the audio input signal as input of the linear regression model.

[0027] In an embodiment, in order to obtain the reverberation send gain, the reverberation gain determiner 110 may, for example, be configured to determine the reverberation send gain as the reverberation gain information by mapping an estimate of the perceived intensity of the reverberation in the audio input signal according to a mapping function. The equivalent reverberation intensity (=perceived intensity after applying the scaling value) may, for example, be fed into the mapping function. For example, the estimate of the perceived intensity of the reverberation is mapped to the listening test data, and then the corresponding fitting curve may, for example, be used for the conversion of the reverberation send gain.

[0028] According to an embodiment, one or more characteristics of the audio input signal may eg depend on an inter-channel correlation of at least one of the one or more subbands of two audio channels of the one or more audio channels of the audio input signal.

[0029] In an embodiment, the one or more characteristics of the audio input signal may, for example, depend on a spectral flatness measure of at least one of the one or more subbands of one or more audio channels of the audio input signal.

[0030] According to an embodiment, the reverberation gain determiner may, for example, be configured to determine an estimate of the perceived intensity of the reverberation in the audio input signal by adopting the model. The reverberation gain determiner may, for example, be configured to determine one or more scaling factors based on one or more features of the audio input signal. Furthermore, the reverberation gain determiner may, for example, be configured to determine the reverberation gain information based on the estimate of the perceived intensity of the reverberation and based on the one or more scaling factors.

[0031] In an embodiment, the one or more scaling factors may, for example, be dependent on an inter-channel correlation of at least one of the one or more sub-bands of two audio channels of the one or more audio channels of the audio input signal.

[0032] According to an embodiment, the one or more scaling factors may, for example, be dependent on the presence of transient signal components in at least one of the one or more audio channels of the audio input signal.

[0033] In an embodiment, the one or more scaling factors may be, for example, based on a spectral transient measure of at least one of the one or more audio channels of the audio input signal. The spectral transient measure may be defined, for example, according to:

[0034] STM(m,k)=max(Z(m,k)-f s (Z(m, k)), 0)

[0035] in

[0036]

[0037] Where m indicates a time index, where k indicates a frequency index, and where b k,i indicates a frequency bin among one or more frequency bins in frequency band / subband k, where Y d (m,b k,i ) indicates one or more frequency bins b in a frequency band k of the at least one audio channel of the one or more audio channels or a combination of the at least one audio channel of the one or more audio channels k,i The amplitude coefficient with time index m, and where f s () indicates recursive averaging and smoothing of the subband signal along time.

[0038] According to an embodiment, the signal processor 120 may be configured to generate a preprocessed audio signal by de-reverberating the audio input signal to attenuate the original reverberation signal component of the audio input signal. The signal processor 120 may be configured to obtain an audio output signal by adding artificial reverberation to the preprocessed audio signal according to the reverberation gain information.

[0039] In an embodiment, the signal processor 120 may be configured to generate a preprocessed audio signal by temporally smoothing the audio input signal. The signal processor 120 may be configured to obtain an audio output signal by adding artificial reverberation to the preprocessed audio signal according to the reverberation gain information.

[0040] According to an embodiment, the signal processor 120 may, for example, be configured to adjust the amount of temporal smoothing according to changes in the audio input signal.

[0041] In an embodiment, the signal processor 120 may, for example, be configured to adjust the amount of temporal smoothing according to changes in the audio input signal.

[0042] According to an embodiment, the signal processor 120 may, for example, be configured to adjust the amount of temporal smoothing according to changes in the loudness of the audio input signal.

[0043] In an embodiment, the signal processor 120 may, for example, be configured to adjust the amount of temporal smoothing according to changes in one or more characteristics of the audio input signal.

[0044] According to an embodiment, the device may, for example, include a ring buffer for real-time processing, for example a ring buffer with a high length and / or a high overlap, which may, for example, be configured to receive an audio input signal or a preprocessed audio signal. The signal processor 120 may, for example, be configured to process the audio input signal or the preprocessed audio signal in the ring buffer to obtain an audio output signal.

[0045] In the following, specific embodiments are described. For example, an algorithm proposed according to an embodiment may first be outlined, and then listening tests and results including statistical evaluations may be provided. In addition, a model for predicting perceived reverberation intensity according to an embodiment is provided, post-processing using scaling factors according to an embodiment is introduced, a real-time application of a trained model according to an embodiment is described, and a conversion to reverberation send gain is introduced.

[0046] In some embodiments, for example, the perceived intensity of reverberation in an audio signal may be estimated, and for example, the level of an artificial reverberation signal may be controlled, such that the artificial reverberation output signal may, for example, have similar reverberation characteristics as the corresponding input signal. The estimation may, for example, employ a linear regression model with sub-band inter-channel coherence and spectral flatness measures as input features, trained using listening test data. For adaptation (e.g. for application of scaling factors), the artificial reverberation control signal may, for example, be calculated based on the time modulation characteristics and / or, for example, based on the correlation between the input channel signals, and may, for example, be applied to calculate an equivalent reverberation level. For example, the resulting quantity may be post-processed using a signal adaptive integral. These concepts may, for example, be applied to control a reverberation send gain (reverberation send gain: e.g., the gain of the (send) (artificial) reverberation) of an artificial reverberation for sound reproduction in a car.

[0047] Reverberation is a very complex element in acoustics and audio signal processing, and embodiments focus on its perceived strength and its control, for example by adjusting the reverberation send gain. Some embodiments may for example be implemented independently of other aspects, such as the frequency-dependent reverberation time of the impulse response or its correlation over channels. Other embodiments may for example take these aspects into account.

[0048] Figure 2 A schematic overview of a specific embodiment is shown. The first stage may, for example, include de-reverberation and extraction of audio features. Specifically, Figure 2 A block diagram of a reverberation control algorithm according to an embodiment is shown. The eigenvalues ​​are input to a model for predicting the perceived reverberation intensity in an input signal. The model employs linear regression and is trained using listening test data. The model output is post-processed with a signal adaptive scaling factor that is manually tuned relative to the characteristics of the newly added artificial reverberation to account for changes in the interaction of the excitation signal and the reverberation due to the different characteristics of the artificial reverberation and the "original" reverberation. It should be noted that data-driven modeling of the perceived reverberation intensity of the artificial reverberation (as is done for the original reverberation) would require listening test data for multiple parameter settings and would not be able to adjust when these settings change. Therefore, this approach is not preferred.

[0049] The equivalent reverberation level is post-processed in real time using signal adaptive time integration. The result is used to calculate a reverberation send gain using a function adapted to a preferred send gain adjusted by an expert listener. The calculated reverberation send gain controls the level of artificial reverberation applied to the dereverberated input signal. De-dereverberation is applied to attenuate the reverberant signal component of the input, since adding artificial reverberation to an input signal with a large amount of "original" reverberation can result in a loss of clarity or create a perception of too much reverberation or produce an aesthetically undesirable superposition of two reverberant signals.

[0050] The results of the listening tests are described below.

[0051] In the listening test, 27 loudness-normalized audio signals with durations between 4.8s and 14.7s were presented. The items selected had small variations in spatial cues (amount of reverberation and energy distribution in the stereo panorama). The set of items ranged from very dry to very reverberant and included recordings of various musical genres as well as recordings of solo instruments and dry speech recordings.

[0052] Participants were asked to rate the perceived reverberation intensity and ensemble width by adjusting sliders on a discrete unipolar scale from 1 to 9. The scale had evenly spaced markings of "very low", "low", "medium", "high" and "very high". An additional characteristic, ensemble width, was provided to make it easier for listeners to judge reverberation intensity independently of the stereo panorama. Testing began with a training session, where three stimuli were presented to give examples of, for example, low reverberation amount and high ensemble width, and vice versa. 15 listeners participated in the test.

[0053] Figure 3 The average rating for each listening test item and the overall average with a 95% confidence interval according to an embodiment are shown. In particular, Figure 3 The mean and 95% confidence interval for the reverberation ratings are shown. The confidence intervals are obtained according to the following formula

[0054]

[0055] in is the estimated mean, t α / 2,n-1 is of the order of α / 2 (where α = 0.05) of the t distribution with n-1 degrees of freedom, is the estimated standard deviation and n is the sample size [8].

[0056] Hereinafter, a linear regression model according to an embodiment is introduced.

[0057] In the following, a computational model for predicting perceived reverberation intensity of an audio signal and its training according to certain embodiments is described.

[0058] First, a model description according to an embodiment is provided.

[0059] Applying a linear regression model to estimate perceived reverberation intensity As input feature x k A linear combination of

[0060]

[0061] With model coefficient β k , bias value β0 and the number of input features K.

[0062] Now, audio feature extraction according to the embodiment is described.

[0063] The input of the linear regression model is the sub-band inter-channel coherence (ICC) and spectral flatness measure (SFM) calculated from the short-term Fourier transform (STFT) coefficients. As shown in Table 1, ICC is calculated in 5 bands and SFM is calculated in 4 bands.

[0064] Table 1 shows the frequency band separation of ICC and SFM according to an embodiment.

[0065]

[0066] (Table 1)

[0067] The STFT coefficients are calculated from an audio signal sampled at 48kHz, with a frame size of 1024 samples, a jump length of 512 samples, and no zero padding. STFT frames with loudness below -65LUFS (loudness units relative to full scale) are removed so that the model training is not affected by silent periods (e.g., speech pauses). The ICC is calculated from the two channel signals of a two-channel stereo input signal according to the following formula [9]

[0068]

[0069] Y 1,1 (m,k) and Y 2,2 (m, k) are the automatic power spectral density (PSD) of the left and right channels respectively, and Y 1,2 (m,k) is the cross PSD, where all PSDs are accumulated over the frequency bins corresponding to the kth frequency band at time index m. k (i,1) indicates the i-th complex spectrum bin of the k-th frequency band of the first audio channel of the audio input signal, and b k (i,2) indicates the i-th complex spectrum bin of the k-th frequency band of the second audio channel of the audio input signal, then Y 1,1 (m,k),Y 12 (m,k) and Y 22 (m,k) can be defined as follows, for example:

[0070]

[0071] b k (m,i,1) * Instructions k(m,i,1), the complex conjugate of, and b k (m,i,2) * Instructions k The complex conjugate of (m,i,2), with time index m. N may, for example, indicate a frequency bin b in frequency band k. k The number of squared STFT magnitude coefficients of the downmix.

[0072] For example, in certain embodiments, a frequency band may, for example, include one or more frequency bins for each audio channel (e.g., in certain embodiments, a frequency band may, for example, be only a single bin). For example: In certain embodiments, for example, formula (3) may be used to obtain the inter-channel correlation of the frequency band.

[0073] SFM (see

[10] ) is calculated as the ratio of the geometric mean to the arithmetic mean according to the following formula:

[0074]

[0075] and

[0076] X(m, k, i) = g(Y d (m,b k,i ) 2 ) (5)

[0077] where g() is a compression function, such as logarithm or square root, or y=x 0.25 , and Y d (m,b k,i ) 2 is the i-th frequency bin b in a frequency band k consisting of N bins k,i The squared STFT magnitude coefficients of the downmix of the left and right input channels of , with the frequency bin time index being m. (Thus, N may, for example, indicate a frequency bin b in frequency band k. k,i ) For example, in an embodiment with two audio channels, the squared STFT amplitude coefficient Y of the downmix d (m,b k,i ) 2 This can be done, for example, by calculating the STFT amplitude coefficient Y of the downmix d (m,b k,i ) is squared to generate, and the downmixed STFT amplitude coefficient Y d (m,b k,i ) can be generated, for example, by combining (eg averaging) the amplitude coefficients of the two audio channels of the frequency bins of the kth frequency band.

[0078] In an embodiment, features are extracted for each frame (eg, for each block), and then the arithmetic mean and standard deviation are calculated to obtain a single value per block, which is then used in the training of the regression model.

[0079] For example, each block may, for example, include audio data for replaying one second of an audio recording. A block may, for example, include spectral data for 94 time points (e.g., 94 frames, assuming that in a particular embodiment, a frame may, for example, have a duration of 21.3 milliseconds, e.g., approximately 94 frames / second when the frames have a 50% overlap). In other embodiments, a block may, for example, include spectral data for any other number of time points. In an example, the spectrum may, for example, be divided into five frequency bands and / or, for example, into four frequency bands, e.g., as shown in Table 1.

[0080] For example, considering the ICC, an ICC value is determined for each of the, for example, five frequency bands of, for example, Table 1, and for each of the, for example, 94 time points. For each of the, for example, five frequency bands, the arithmetic mean of the, for example, 94 ICCs of the block is determined, and thus, for example, five arithmetic mean ICC values ​​for the five frequency bands can be obtained. And / or, for each of the, for example, five frequency bands, the standard deviation of the, for example, 94 ICCs of the block is determined, and thus, for example, five standard deviation ICC values ​​for the five frequency bands can be obtained. Thus, for example, 10 ICC-related eigenvalues ​​are generated for the, for example, five frequency bands of the block.

[0081] For example, considering SFM, for each of the four frequency bands of Table 1, for example, and for each of the 94 time points, for example, an SFM value is determined. For each of the four frequency bands, for example, an arithmetic mean of the 94 SFMs of the block is determined, and thus, for example, four arithmetic mean SFM values ​​of the four frequency bands can be obtained. And / or, for each of the four frequency bands, for example, a standard deviation of the 94 ICCs of the block is determined, and thus, for example, four standard deviation SFM values ​​of the four frequency bands can be obtained. Thus, for example, for the four frequency bands of the block, for example, 8 eigenvalues ​​related to SFM are obtained.

[0082] Regarding the combination of the ICC sample and the SFM sample, for example, 18 feature values ​​for one block can be obtained.

[0083] Now, model training according to an embodiment is described.

[0084] Processing short data blocks is crucial for real-time processing (especially for non-stationary signals) as well as increasing the available amount of training data. Each item was windowed using a training block length of 6 s and zero overlap. Items shorter than 12 s in length were processed as a single block over their entire length. Performance when training with overlapping blocks was also evaluated, but no improvement in model accuracy was observed.

[0085] The training dataset was extended by adding second excerpts with similar spatial properties for some of the songs used in the listening test. Additionally, some key items (for which predictions were insufficient) and reference ratings provided by an expert listener were added. To ensure that the model's predictions for dry and clean speech signals were small, recordings of male and female speakers were added to the dataset and the reference annotations were set to -1 MOS (mean opinion score). The dataset was extended and multiple blocks were used for each item, resulting in a total of 100 observations.

[0086] To account for the nonlinear relationship between the listening test data and the eigenvalues, the mean and standard deviation of the eigenvalues ​​were exponentiated. The exponents were determined by manual adjustment based on the evaluation of residual plots, which revealed nonlinear relationships that could not be modeled by linear regression.

[0087] The model is trained using the ordinary least squares (OLS) algorithm, which makes the prediction The mean squared error between the mean rating obtained in the listening test and the mean rating obtained in the listening test is minimized. Evaluation is performed using leave-one-out cross-validation

[11] , where the model is trained on all observations except for one item and evaluated on the left-out item. This process is repeated for each item.

[0088] Now, feature selection according to the embodiment is described.

[0089] The training is started with all available values, such as the arithmetic mean and standard deviation of each sub-band feature. The sub-band feature can be, for example, the ICC or SFM of the sub-band (frequency band).

[0090] In order to avoid overfitting and reduce computational load, unhelpful independent variables were removed one by one by evaluating the p-value. A low p-value indicates that there is probably no relationship between the corresponding eigenvalue and the listening test results (see

[11] ).

[0091] For example, in the above example with 18 eigenvalues ​​(10 ICC-related eigenvalues ​​and 8 SFM-related eigenvalues), stepwise removal of all irrelevant eigenvalues ​​may, for example, result in a model with 4 input variables (e.g., the other 14 irrelevant or less relevant input variables / eigenvalues ​​may be stepwise removed, for example, by performing a multiple regression analysis (see, e.g.,

[11] ). Figure 4 The performance of the model is shown.

[0092] Specifically, Figure 4The ratio of the predicted reverberation intensity to the observed reverberation intensity according to an embodiment is shown. The predictions for the training block of listening test items are shown with dots, and the predictions obtained for the training blocks of the extended data set (items from the extended data set) are represented by triangles. The average prediction for each item is represented by a cross. Marks representing the same block of items have the same listening test rating. The chart shows a good correlation between the predictions and observations, with a correlation coefficient of 0.75 and an average error of 1.27 MOS. For listening test ratings around 3 MOS, there is a trend of overestimation, while the high scores are slightly underestimated.

[0093] In the following, a reverberation send gain calculation according to an embodiment is provided. In particular, the calculation of a preferred reverberation send gain of an artificial reverberation given the perceived strength of the original reverberation in the input signal is described. Since the original reverberation (of the input signal) and the secondary (artificial) reverberation are not matched, equal reverberation levels result in different perceived reverberation strengths. Therefore, the reverberation send gain cannot be directly calculated based on the given perceived reverberation strength.

[0094] Data-driven modeling of the perceived intensity of artificial reverberation is not feasible because it requires subjective data for different reverberation settings.

[0095] In an embodiment, a manually adjusted scaling factor s is introduced t and c , to compensate for signal-dependent effects, e.g., as described below. For the described processing, the intermediate amount, equivalent reverberation level is defined as

[0096]

[0097] The equivalent reverberation level represents the desired level of artificial reverberation for a given input signal, which results in similar reverberation strengths for the artificial and original reverberations. The equivalent reverberation level is then converted into a reverberation send gain by a mapping function, which is determined in a separate tuning experiment.

[0098] Now, aspects of the equivalent reverberation level according to the embodiment are described.

[0099] Signals with strong transients, such as drums, result in higher perceived reverberation intensity and are less effective at masking reverberation than stationary signals.

[0100] Artificial reverberation with a larger time constant may require a lower reverberation level to induce a similar intensity perception as the original reverberation. Some embodiments provide a novel spectral transient measure (STM) for quantifying the intensity of transients in a signal, such as

[0101] STM(m,k)=max(Z(m,k)-fs (Z(m,k)),0) (7)

[0102] and

[0103]

[0104] f s () indicates recursive averaging along time and smoothing of the subband signal.

[0105] For example, in a particular embodiment, the frequency bands may be, for example, the frequency bands in the left column of Table 1 or the frequency bands in the right column of Table 1. Alternatively, other frequency bands / band configurations may be employed. Each frequency band may, for example, include all time-frequency bins with a time index m within the frequency band.

[0106] A single STM value is obtained by averaging the STM values ​​in the frequency bands from 23 to 70 Hz and 5.2 to 21 kHz. This frequency band selection focuses on transients caused by percussion instruments, while transients of other instruments (such as piano) are less considered.

[0107] Scaling factor t According to STM calculation,

[0108]

[0109] in represents the average value of the STM value of a block of length 1s, which is then clipped according to the following formula

[0110] S t,min ≤S t ≤S t,max (10)

[0111] m t 、n t 、s t,min and t,max It is a tunable parameter that is adjusted according to the characteristics of artificial reverberation.

[0112] If the reverberation time of the "original" reverberation exceeds the reverberation time of the artificial reverberation, a larger equivalent reverberation level is required. The second scaling factor s c Designed to adapt the estimated reverberation strength to input signals with long reverberation times and high reverberation volumes (e.g. classical music). It is assumed that these terms are fairly decorrelated due to the large diffuse sound components. c is calculated as

[0113]

[0114] in is the average of the ICCs of all bands and frames for a block of length 1s and is then clipped according to

[0115] S c,min ≤S c ≤S c,max (12)

[0116] m c 、n c 、s c,min and c,max It is an adjustable parameter.

[0117] In the following, signal adaptive time integration according to embodiments is described.

[0118] In order to control the reverberation send gain in real time, it is necessary to react to changes in the input signal while not introducing an obvious adjustment of the reverberation level. To solve this problem with low latency and low computational load, the predictions are computed in blocks of length 8s, with 7s overlap per block, so that new predictions are computed at a rate of 1Hz. Signals with low input levels (e.g. below -65LUFS) are not fed to the model as during training. The equivalent reverberation level is calculated using recursive averaging of a single-pole IIR filter. (6) Perform time smoothing,

[0119]

[0120] With a smoothing coefficient α.

[0121] To avoid excessive reverberation levels at transitions between input signals with very different characteristics, trajectory changes are detected and a fast adaptation of the equivalent reverberation level is implemented. The fast adaptation is performed by reducing the model block length to 1s and increasing the smoothing factor α. For detection, changes in the STM variance are identified, as changes in transient quantities are good indicators, such as the transition from music to speech, where lower reverberation levels are desired.

[0122] Figure 5 shows real-time processing of an explanatory input file according to an embodiment, which includes 25s of classical music and a sudden transition to pop music, which can be observed in the upper graph showing the spectrogram. In particular, Figure 5 The top part shows the spectrogram of the audio signal after 25 seconds of transition from classical music to pop music. Figure 5 The middle part of shows the equivalent reverberation level before (solid line) and after (dashed line) time smoothing. Figure 5 The bottom of shows the smoothing factor (solid line) and the total scaling value (dashed line).

[0123] The middle figure shows the equivalent reverberation level before (solid line) and after (dashed line) time smoothing, with values ​​of around 7 MOS for classical music and 2.2 MOS for pop projects. The lower sub-figure shows the product of the scaling values, called the total scaling value, as a dashed line (s t ·s c ). Due to the low correlation of the classical music project, the total scaling value is above 1 until the track changes. The increase in transient volume at t=25s triggers the track change detection and reduces the total scaling value to about 0.7.

[0124] The smoothing coefficient α, plotted with a solid line in the lower figure, decreases after an adaptation phase of 11 s, since the loudness and the equivalent reverberation level are fairly stable. The detection of a trajectory change at t=25 s temporarily increases α and leads to a fast adaptation of the equivalent reverberation level.

[0125] Now, conversion from reverberation prediction to reverberation send gain according to an embodiment is described.

[0126] For example: As a last step, the send gain of the reverberation processor can be calculated, for example, given the equivalent reverberation level. For this purpose, a second listening test was conducted in which 5 expert listeners adjusted the send gains of two artificial reverberations simulating the acoustic environments of a concert hall (T60=2.2s) and a jazz club (T60=1.3s) in a car according to their preferences.

[0127] Figure 6 1 shows the reference reverberation transmit gain on the equivalent reverberation level of the fitted polynomial according to an embodiment. In particular, Figure 6 The average send gain on the equivalent reverberation level obtained according to equation (1) is plotted, along with the 95% confidence intervals for all 30 test items for both artificial reverberations. Since longer reverberation tails lead to less masking and higher perceived reverberation intensity, while transients require lower reverberation send gains, the adjustable parameters in equations (9) and (11) are adjusted so that strong transients and low correlations result in a lower equivalent reverberation level for the concert hall than for the jazz club. Therefore, a single mapping curve can be used for both reverberation curves. The Spearman rank correlation coefficient is 0.81, confirming the monotonic relationship between the two variables, which can be approximated by a third-order polynomial with low error.

[0128] Gain adjustment is used to fit the polynomial mapping,

[0129]

[0130] like Figure 6 As shown, it will smooth the equivalent reverberation level Converts to the predicted reverb send gain g.

[0131] In the following, the evaluation aspect is considered.

[0132] To evaluate the polynomial mapping, the leave-one-out cross-validation technique was used, which means that all pairs of values ​​except one were used to fit the polynomial function and the last data pair was used to obtain the error for that particular data point. This process was repeated for all listening test items and the mean absolute error (MAE) was 1.74 dB. According to [7], this deviation from the preferred reverberation send gain leads to a negligible decrease in aesthetic quality. The limits of the 95% confidence interval obtained according to equation (1) are located between 1.27 dB and 2.21 dB. The absolute maximum error corresponds to 5.11 dB and the correlation between the predicted and observed values ​​is located at 0.89.

[0133] Figure 7 The equivalent reverberation level at the reverberation send gain error according to an embodiment is shown. Figure 7 The scatter plot depicted in Figure 1 shows the reverb send gain error at the equivalent reverb level, revealing that the error is evenly distributed over the entire value range with only one outlier at 2.6MOS.

[0134] Concepts for controlling the level of artificial reverberation have been provided. Some of the concepts provided may, for example, employ a model for predicting the perceived reverberation intensity based on a linear regression model trained using listening test data. The method employs artificial adjustment of various parameter settings of the artificial reverberation. Fast adaptation to changes in input signal characteristics is achieved by signal adaptive time integration. The algorithm was evaluated in an in-car sound reproduction application and was able to predict a preferred reverberation send gain with a MAE of 1.74 dB.

[0135] Extensions to the embodiments presented above are now described.

[0136] For example, other application scenarios may be implemented.

[0137] Embodiments may be applied, for example, to control reverberation in a variety of other application scenarios. In two-channel stereophonic reproduction, artificial reverberation is applied to support the sense of externalization, e.g., the listener localizes sound events, objects, and sources outside the head. This is in contrast to standard sound reproduction using headphones, where typically sound is perceived as coming from inside the head.

[0138] Artificial reverberation is applied to multiple sound sources and mixtures of single sound sources in music production. The proposed method can also be extended to predict the perceived amount of reverberation of a single sound source or a single component in a mixed signal (e.g. a group of sound sources). Information about the amount of each single component can be used to automatically control or artificial reverberation in sound production (music production, movie sound, podcasts and other user-generated content).

[0139] Other embodiments may, for example, implement alternative models.

[0140] The linear model can be replaced, for example, by other data-driven models, such as by any function

[0141]

[0142] where the scalar output is calculated from the input vector x and the model parameters Θ By optimizing The model parameters are determined by a criterion (e.g., minimizing the mean squared error) between the training set and the target output y. Another well-known class of such trainable models is artificial neural networks, in particular deep neural networks (DNNs). It should be noted that the larger the set of parameters θ, the more data is needed to train the model, so that the performance is good not only for the training data, but also for new data that was not used during training. is also good. This property is called generalization. However, other models such as DNNs can lead to better predictions (with higher accuracy) when more training data is available.

[0143] Such DNNs are implemented by combining layers of processing units, where the units themselves have trainable parameters. Commonly used layer types are convolutional layers, dense layers (also called fully connected layers), and recurrent layers. They differ in what type of units are implemented and how these units are connected to each other. Other types of layers are used to support the training process, where training refers to the process of optimizing the parameters Θ through numerical optimization.

[0144] In the following, further embodiments are provided.

[0145] An apparatus and / or method is provided for controlling the level of an artificially generated reverberation signal to be added to an audio signal using a model of the perceived intensity of reverberation in an original input signal.

[0146] According to an embodiment, the input signal may be dereverberated, for example, to attenuate the original reverberant signal component.

[0147] In an embodiment, the linear regression model may, for example, take different audio feature values ​​as input.

[0148] According to an embodiment, sub-band SFM and / or sub-band ICC may, for example, be used as input audio features.

[0149] In an embodiment, adjustable scaling values ​​for changing the model predictions may be employed, for example, these scaling values ​​depending on the audio feature values ​​to compensate for the interaction of the artificial reverberation and the change of the dereverberated input signal compared to the original reverberation and the direct signal.

[0150] According to an embodiment, for example, an adjustable scaling value according to sub-band ICC and sub-band STM may be adopted.

[0151] In an embodiment, for example, a circular buffer with high overlap may be employed for real-time processing.

[0152] According to an embodiment, adaptive temporal smoothing may be used, for example, to adjust the amount of smoothing according to changes in the input signal.

[0153] In an embodiment, the changes in loudness and model predictions may, for example, be evaluated to control a smoothing factor of the time-recursive averaging.

[0154] According to an embodiment, a trajectory change may be detected, for example, based on a change in the variance of the audio feature to temporarily reduce a smoothing coefficient of the time recursive averaging and reset the circular buffer.

[0155] In an embodiment, a mapping function suitable for listening test data may be employed to convert the tuned model predictions into reverberation send gains, for example.

[0156] The proposed concept can be applied, for example, to sound reproduction inside a car to simulate an acoustic environment of larger size and pleasing sounding spatial characteristics. The simulated acoustic environment is achieved by processing the audio input signal so that the reverberant signal component of the output signal is perceptually similar to the reproduction of the direct signal component in the new environment. The implementation has low latency and low workload.

[0157] Although some aspects have been described in the context of devices, it is clear that these aspects also represent the description of corresponding methods, where modules or devices correspond to method steps or features of method steps. Similarly, the various aspects described in the context of method steps also represent the description of corresponding modules or items or features of corresponding devices. Some or all of these method steps can be performed by (or using) hardware devices (e.g., microprocessors, programmable computers, or electronic circuits). In some embodiments, such devices can perform one or more of the most important method steps.

[0158] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software or at least in part in hardware or at least in part in software. The implementation may be performed using a digital storage medium (e.g., a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory) on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0159] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0160] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is used to perform one of the methods. The program code can be stored on a machine-readable carrier, for example.

[0161] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0162] In other words, an embodiment of the inventive method is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0163] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium, or the recorded medium are typically tangible and / or non-transitory.

[0164] Therefore, another embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or this sequence of signals may, for example, be configured to be transmitted via a data communication connection (eg via the Internet).

[0165] Another embodiment comprises a processing device, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0166] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0167] Another embodiment according to the invention comprises a device or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The device or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0168] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0169] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0170] The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0171] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the scope of the appended claims rather than the specific details presented by the description and explanation of the embodiments herein be limited.

[0172] References

[0173] [1]Paulus,J.,Uhle,C.,and Herre,J., "Perceived Level of LateReverberation in Speech and Music," in Proc.AES130th Conv., 2011.

[0174] [2]Gardner, WG and Griesinger, D., "Reverberation level matching experiments," in Proc. Sabine Centennial Symposium, Acoust. Soc. of Am., 1994.

[0175] [3] Leonard, B., King, R., and Sikora, G., "The Effect of Playback System onReverberation Level Preference," in Audio Engineering Society Convention 134, 2013.

[0176] [4] Leonard, B., King, R., and Sikora, G., "The Effect of AcousticEnvironment on Reverberation Level Preference," in Audio Engineering Society Convention 133, 2012.

[0177] [5]Sarroff,A.M.and Bello,J.P.,“Toward a Computational ModelofPerceived Spaciousness in Recorded Music,”J.Audio Eng.Soc,59(7 / 8),pp.498–513,2011.

[0178] [6]Uhle,C.,Paulus,J.,and Herre,J.,“Predicting the PerceivedLevelofLate Reverberation Using Computational Models of Loudness,”inProc.17thInt.Conf.on Digital Signal Process.(DSP),2011.

[0179] [7]Paulus,J.,Uhle,C.,Herre,J.,and M.,“A Study on thePreferredLevel of Late Reverberation in Speech and Music,”in Proc.of the60thInt.Conf.of Audio Eng.Soc.,2016.

[0180] [8]Gilleland,E.,Confidence Intervals for Forecast Verification,National Center For Atmospheric Research,Boulder,Colorado,2010.

[0181] [9]Allen,J.,Berkeley,D.,and Blauert,J.,“Multimicrophone Signal-Processing Technique to Remove Room Reverberation from Speech Signals,”J.Acoust.Soc.Am.,62,1977.

[0182]

[10] Gray,A.H.and Markel,J.D.,“A Spectral-Flatness Measure forStudyingthe Autocorrelation Method of Linear Prediction of Speech Analysis,”IEEETrans.Acoust.,Speech,and Sig.Process.,22,pp.207–217,1974.

[0183]

[11] James,G.,Witten,D.,Hastie,T.,and Tibshirani R.,editors,“AnIntroduction to Statistical Learning:with Applications in R,”number 103inSpringer Texts in Statistics,Springer,New York,2013.

Claims

1. A device for processing an audio input signal comprising one or more audio channels to obtain an audio output signal, wherein the device comprises: A reverberation gain determiner (110) is configured to determine reverberation gain information according to an audio input signal, and The signal processor (120) is configured to obtain an audio output signal by adding artificial reverberation to the audio input signal or adding artificial reverberation to the preprocessed audio signal according to the audio input signal according to the reverberation gain information.

2. The device according to claim 1, The reverberation gain determiner (110) is configured to determine reverberation gain information based on an estimate of a perceived strength of reverberation in the audio input signal.

3. The device according to claim 2, The reverberation gain determiner (110) is configured to determine the reverberation gain information by employing a model that returns an estimate of the perceived strength of reverberation in the audio input signal upon receiving information about one or more characteristics of the audio input signal.

4. The device according to claim 3, The model adopted by the reverberation gain determiner (110) is a linear regression model that uses one or more feature values ​​of one or more features of the audio input signal as input of the linear regression model.

5. The device according to claim 3 or 4, The reverberation gain determiner (110) is configured to determine the reverberation send gain as the reverberation gain information by mapping an estimate of the perceived intensity of the reverberation in the audio input signal according to a mapping function to obtain a reverberation send gain.

6. The device according to any one of claims 3 to 5, The one or more features of the audio input signal are based on an inter-channel correlation of at least one of the one or more sub-bands of two audio channels of the one or more audio channels of the audio input signal.

7. The device according to any one of claims 3 to 6, The one or more characteristics of the audio input signal are measured based on a spectral flatness measure of at least one sub-band of one or more sub-bands of one or more audio channels of the audio input signal.

8. The device according to any one of claims 3 to 7, wherein the reverberation gain determiner is configured to determine an estimate of the perceived strength of the reverberation in the audio input signal by adopting a model, wherein the reverberation gain determiner is configured to determine one or more scaling factors based on one or more characteristics of the audio input signal, and The reverberation gain determiner is configured to determine the reverberation gain information based on an estimate of the perceived strength of the reverberation and based on one or more scaling factors.

9. The device according to claim 8, The one or more scaling factors are based on an inter-channel correlation of at least one of the one or more sub-bands of two audio channels of the one or more audio channels of the audio input signal.

10. The device according to claim 8 or 9, The one or more scaling factors are dependent on the presence of a transient signal component in at least one of the one or more audio channels of the audio input signal.

11. The device according to any one of claims 8 to 10, wherein the one or more scaling factors are based on a spectral transient measure of at least one of the one or more audio channels of the audio input signal, where the spectral transient measurement is defined as follows: STM(m,k)=max(Z(m,k)-f s (Z(m, k)), 0) in Where m indicates the time index, where k indicates the frequency index, where b k,i indicates one frequency bin in one or more frequency bins in frequency band / sub-band k, where Y d (m,b k,i ) indicates one or more frequency bins b in a frequency band k of the at least one audio channel of the one or more audio channels or a combination of the at least one audio channel of the one or more audio channels k,i The amplitude coefficient with time index m, and where f s () represents recursive averaging along time and smoothing of the subband signal.

12. The device according to any one of the preceding claims, wherein the signal processor (120) is configured to generate a preprocessed audio signal by de-reverberating the audio input signal to attenuate an original reverberation signal component of the audio input signal, and The signal processor (120) is configured to obtain an audio output signal by adding artificial reverberation to the preprocessed audio signal according to the reverberation gain information.

13. The device according to any one of the preceding claims, The signal processor (120) is configured to generate a preprocessed audio signal by time-smoothing the audio input signal, and The signal processor (120) is configured to obtain an audio output signal by adding artificial reverberation to the preprocessed audio signal according to the reverberation gain information.

14. The device according to claim 13, The signal processor (120) is configured to adjust the amount of temporal smoothing based on changes in the audio input signal.

15. The device according to claim 13 or 14, The signal processor (120) is configured to adjust the amount of temporal smoothing based on changes in the audio input signal.

16. The device according to claim 15, The signal processor (120) is configured to adjust the amount of temporal smoothing according to changes in the loudness of the audio input signal.

17. The apparatus according to claim 15 or 16, further according to claim 3, The signal processor (120) is configured to adjust the amount of temporal smoothing based on changes in the variance of one or more features of the audio input signal.

18. Apparatus according to any one of the preceding claims, The device includes a ring buffer for receiving an audio input signal or a pre-processed audio signal, and The signal processor (120) is configured to process the audio input signal or the pre-processed audio signal in the ring buffer to obtain an audio output signal.

19. A method for processing an audio input signal to obtain an audio output signal, the method comprising: determining reverberation gain information based on the audio input signal, and The audio output signal is obtained by adding artificial reverberation to the audio input signal or adding artificial reverberation to the preprocessed audio signal according to the audio input signal according to the reverberation gain information.

20. A computer program for implementing the method according to claim 19 when executed on a computer or a signal processor.