Method for processing audio signal, signal processing unit, binaural renderer, audio encoder, and audio decoder

Abstract

To provide a method for processing an audio signal (504) in accordance with room impulse response.SOLUTION: An audio signal (504) is processed (502) by using an early part of room impulse response separate from late reverberation of the room impulse response, where processing (514) of the late reverberation comprises generating a scaled reverberated signal, scaling (526) being dependent on the audio signal (504). The processed early part (506) of the audio signal (504) and the scaled reverberated signal are combined.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to the field of audio encoding / decoding, in particular to the field of spatial audio coding and spatial audio object coding, e.g. 3D audio codec systems. Embodiments of the present invention relate to a method for processing an audio signal according to a room impulse response, to a signal processing unit, a binaural renderer, an audio encoder and an audio decoder. [Background technology]

[0002] Spatial audio coding tools are well known in the art and are standardized, for example, in the MPEG Surround standard. Spatial audio coding starts with multiple original inputs, for example, five or seven input channels, which are identified by their arrangement in the playback setup, for example, as left channel, center channel, right channel, left surround channel, right surround channel, and low frequency extension channel. The spatial audio encoder may derive one or more downmix channels from the original channels and may further derive parametric data related to spatial cues, such as inter-channel level differences, inter-channel phase differences, inter-channel time differences of channel coherence values. To finally obtain output channels that are approximate versions of the original input channels, the one or more downmix channels are transmitted to a spatial audio decoder along with parametric side information indicative of spatial cues to decode the downmix channels and the associated parametric data. The arrangement of the channels in the output setup may be fixed, for example, 5.1 format, 7.1 format, etc.

[0003] Also spatial audio object coding tools are well known in the art and are standardized, for example, in the MPEG SAOC standard (SAOC = Spatial Audio Object Coding). In contrast to spatial audio coding, which starts from the original channels, spatial audio object coding starts from audio objects that are not automatically dedicated for a certain rendering playback setup. Instead, the placement of audio objects in the playback scene is flexible and can be set by the user, for example, by inputting certain rendering information into the spatial audio object coding decoder. Alternatively or additionally, the rendering information can be transmitted as additional side information or metadata, which may include information on which audio objects should be placed (for example, over time) at their positions in the playback setup. To obtain a certain data compression, some audio objects are encoded using a SAOC encoder, which calculates one or more transport channels from the input objects by downmixing the objects according to certain downmix information. Furthermore, the SAOC encoder calculates parametric side information representing inter-object cues, such as object level difference (OLD), object coherence values, etc. As in the case of SAC (SAC = spatial audio coding), the inter-object parametric data is calculated for each time / frequency tile. For a frame of an audio signal (e.g., 1024 or 2048 samples), multiple frequency bands (e.g., 24, 32 or 64 bands) are considered so that parametric data is provided for each frame and for each frequency band. For example, when an audio part has 20 frames and each frame is subdivided into 32 frequency bands, the number of time / frequency tiles is 640.

[0004] In a 3D audio system, it may be desired to provide a spatial impression of an audio signal as if the audio signal were listened to in a particular room. In such a situation, a room impulse response of a particular room is provided, for example based on measurements thereof, and used to process the audio signal when presenting it to a listener. In such a presentation, it may be desired to process the direct sound and early reflections separated from the late reverberations. Summary of the Invention [Problem to be solved by the invention]

[0005] The objective underlying the present invention is to provide an approved technique for separately processing an audio signal with the early part and the late reverberation of the room impulse response, thereby making it possible to achieve results that are as perceptually equivalent as possible to the results of convolution of the audio signal with the complete impulse response. [Means for solving the problem]

[0006] This object is achieved by a method according to claim 1, a signal processing unit according to claim 19, a binaural renderer according to claim 23, an audio encoder according to claim 24 and an audio decoder according to claim 25.

[0007] The invention is based on the inventor's discovery that in conventional approaches, problems exist when processing an audio signal with a room impulse response, in that the results of processing the audio signal separately for the initial part and the reverberation deviate from the results when applying convolution with the complete impulse response. The invention is further based on the inventor's discovery that, for example, when using synthetic reverberation approaches, the influence of the input audio signal on the reverberation is not sufficiently preserved, so that the appropriate level of reverberation depends on both the input audio signal and the impulse response. The influence of the impulse response can be taken into account by using known reverberation characteristics as input parameters. The influence of the input signal can be taken into account by signal-dependent scaling to adapt the level of reverberation determined based on the input audio signal. It has been found that with this approach, the perceived level of reverberation better matches the level of reverberation when using a complete convolution approach for binaural rendering.

[0008] (1) The present invention provides a method for processing an audio signal according to a room impulse response, the method comprising: - separately processing an audio signal using an early part and a late reverberation of a room impulse response, where processing the late reverberation comprises generating a scaled reverberation signal, the scaling being dependent on the audio signal; synthesizing the processed audio signal with the initial part of the room impulse response and the scaled reverberation signal; Includes. Compared to the conventional approaches described above, the inventive approach is advantageous because it allows scaling the late reverberation without the need to compute a full convolution result or to apply extensive and inaccurate auditory models. An embodiment of the inventive approach provides an easy way to scale the artificial late reverberation so that it sounds like a full convolution approach. The scaling is based on the input signal, no additional auditory models or target reverberation loudness are required. The scaling factors can be derived in the time-frequency domain, which is an advantage since the audio material in the encoder / decoder chain is often also available in this domain.

[0009] (2) According to an embodiment, the scaling may depend on the state of one or more input channels of the audio signal (e.g., the number of input channels, the number of active input channels, and / or the activity among the input channels). This is advantageous since the scaling can be easily determined from the input audio signal with reduced computational overhead. For example, the scaling can be determined by simply determining the number of channels in the original audio signal to be downmixed to the currently considered downmix channel, which contains a reduced number of channels compared to the original audio signal. Alternatively, the number of active channels (channels showing some activity in the current audio frame) to be downmixed to the currently considered downmix channel can form the basis for scaling the reverberant signal.

[0010] (3) According to an embodiment, the scaling (in addition to or as an alternative to the input channel conditions) depends on a predefined or calculated correlation measure of the audio signal. Using a predefined correlation measure is advantageous since it reduces the computational complexity of the process. The predefined correlation measure may have a fixed value, for example in the range of 0.1 to 0.9, which may be empirically determined based on an analysis of a plurality of audio signals. On the other hand, if it is desired to obtain a more accurate measure for each currently processed audio signal, it is advantageous to calculate the correlation measure despite the additional computational resources required.

[0011] (4) According to an embodiment, generating the scaled reverberation signal includes applying a gain factor, the gain factor being determined based on a state of one or more input channels of the audio signal and / or based on a predefined or calculated correlation measure for the audio signal, and the gain factor may be applied before, during or after processing late reverberation of the audio signal. This is advantageous because the gain factor can be easily calculated based on the above parameters and can be flexibly applied for the reverberators in the processing chain depending on the implementation details.

[0012] (5) According to an embodiment, the gain factor is determined as follows: g=c u +ρ·(c c -c u ) however ρ = a predefined or calculated correlation measure for the audio signal; c u , c c = a factor indicating the state of one or more input channels of an audio signal, c u refers to a totally uncorrelated channel, and c c relates to the channels which are correlated as a whole. This is advantageous because the factor scales over time with the number of active channels in the audio signal.

[0013] (6) According to an embodiment, cu and c c is determined as follows:

number

number

[0014] (7) According to an embodiment, the gain factors may be lowpass filtered across multiple audio frames, and the gain factors may be lowpass filtered as follows:

number

number

number

[0015] (8) According to an embodiment, generating the scaled reverberation signal includes correlation analysis of the audio signal, the correlation analysis of the audio signal may include determining a synthesis correlation measure for audio frames of the audio signal, the synthesis correlation measure may be calculated by synthesizing correlation coefficients for multiple channel synthesis of one audio frame, each audio frame including one or more time slots, and synthesizing the correlation coefficients may include averaging multiple correlation coefficients for the audio frames. This is advantageous because the correlation can be described by one single value that describes the overall correlation of one audio frame - there is no need to deal with multiple frequency dependent values.

[0016] (9) According to an embodiment, determining the composite correlation measure may include (i) calculating an overall average value for each channel of an audio frame, (ii) calculating a zero-mean audio frame by subtracting the average values ​​from the corresponding channels, (iii) calculating a correlation coefficient for the multiple channel composite, and (iv) calculating the composite correlation measure as an average of the multiple correlation coefficients. As mentioned above, this is advantageous because only one single overall correlation value is calculated per frame (an easy process), and the calculation can be done in the same way as the "standard" Pearson's correlation coefficient, which also uses a zero-mean signal and its standard deviation.

[0017] (10) According to an embodiment, the correlation coefficient for the channel combination is determined as follows:

number

[0018] (11) According to an embodiment, processing late reverberation of the audio signal comprises downmixing the audio signal and applying the downmixed audio signal to a reverberator. This is advantageous because, for example, the processing in the reverberator has to deal with fewer channels and the downmix process can be directly controlled.

[0019] (12) The present invention provides a signal processing unit comprising an input for receiving an audio signal, an early part processor for processing the received audio signal in accordance with an early part of a room impulse response, a late reverberation processor for processing the received audio signal in accordance with a late reverberation of the room impulse response, the late reverberation processor being configured or programmed to generate a scaled reverberation signal that is dependent on the received audio signal, and an output for combining the audio signal processed using the early part of the room impulse response and the scaled reverberation signal into an output audio signal.

[0020] (13) According to an embodiment, the late reverberation processor comprises a reverberator that receives an audio signal and generates a reverberation signal, a correlation analyzer that generates a gain factor that is dependent on the audio signal, and a gain stage that is coupled to an input or an output of the reverberator and controlled by the gain factor provided by the correlation analyzer.

[0021] (14) According to an embodiment, the signal processing unit further comprises at least one of a low-pass filter coupled between the correlation analyzer and the gain stage, and a delay element coupled between the gain stage and an adder, the adder being further coupled to the initial partial processor and to the output.

[0022] (15) The present invention provides a binaural renderer comprising the signal processing unit of the present invention.

[0023] (16) The invention provides an audio encoder for coding an audio signal, comprising an inventive signal processing unit for processing the audio signal prior to coding or an inventive binaural renderer.

[0024] (17) The invention provides an audio decoder for decoding an encoded audio signal, comprising an inventive signal processing unit for processing the decoded audio signal or an inventive binaural renderer. Embodiments of the present invention will now be described with reference to the accompanying drawings. [Brief description of the drawings]

[0025] [Figure 1] This shows an overview of the 3D audio encoder of the 3D audio system. [Diagram 2] 1 shows an overview of the 3D audio decoder of the 3D audio system. [Diagram 3] 3 shows an example for implementing a format converter that may be implemented in the 3D audio decoder of FIG. 2 . [Figure 4] 3 illustrates an embodiment of a binaural renderer that may be implemented in the 3D audio decoder of FIG. 2; [Diagram 5] An example of a room impulse response h(t) is shown below. [Figure 6(a)] Different possibilities for processing an audio input signal with a room impulse response are presented, and a complete audio signal is processed according to the room impulse response. [Figure 6(b)] Different possibilities for processing an audio input signal with a room impulse response are presented, showing separate processing of early and late reverberant parts. [Figure 7] 1 shows a block diagram of a signal processing unit, such as a binaural renderer, that operates in accordance with the teachings of the present invention. [Figure 8] 2 illustrates a schematic of binaural processing of an audio signal in a binaural renderer according to one embodiment of the present invention; [Figure 9] 9 illustrates a schematic of processing in a frequency domain reverberator of the binaural renderer of FIG. 8 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Next, an embodiment of the inventive technique will be described. The following description begins with a system overview of a 3D audio codec system in which the inventive technique may be implemented.

[0027] 1 and 2 show algorithmic blocks of a 3D audio system according to an embodiment. More specifically, FIG. 1 shows an overview of a 3D audio encoder 100. The audio encoder 100 receives an input signal, more specifically a number of input channels, which provide the audio encoder 100 with a number of channel signals 104, a number of object signals 106 and corresponding object metadata 108, at a pre-renderer / mixer circuit 102, which may be provided optionally. The object signals 106 (see signal 110) processed by the pre-renderer / mixer 102 may be provided to a SAOC encoder 112 (SAOC=Spatial Audio Object Coding). The SAOC encoder 112 generates a SAOC transport channel 114, which is provided to a USAC encoder 116 (USAC=Unified Speech and Audio Coding). Furthermore, a signal SAOC-SI 118 (SAOC-SI=SAOC Side Information) is also provided to the USAC encoder 116. The USAC encoder 116 further receives an object signal 120 directly from the pre-renderer / mixer, as well as a channel signal and a pre-rendered object signal 122. The object metadata information 108 is applied to an OAM encoder 124 (OAM=Object Metadata), which provides compressed object metadata information 126 to the USAC encoder. The USAC encoder 116 generates a compressed output signal mp4, as shown at 128, based on the above mentioned input signals.

[0028] Fig. 2 shows an overview of a 3D audio decoder 200 of a 3D audio system. The encoded signal 128 (mp4) generated by the audio encoder 100 of Fig. 1 is received in the audio decoder 200, more specifically in a USAC decoder 202. The USAC decoder 202 decodes the received signal 128 into a channel signal 204, a pre-rendered object signal 206, an object signal 208 and a SAOC transport channel signal 210. Furthermore, compressed object metadata information 212 and a signal SAOC-SI 214 are output by the USAC decoder 202. The object signal 208 is provided to an object renderer 216, which outputs a rendered object signal 218. The SAOC transport channel signal 210 is fed to a SAOC decoder 220, which outputs a rendered object signal 222. The compressed object meta information 212 is provided to an OAM decoder 224, which outputs respective control signals to the object renderer 216 and the SAOC decoder 220 to generate the rendered object signal 218 and the rendered object signal 222. The decoder further comprises a mixer 226, as shown in Fig. 2, which receives the input signals 204, 206, 218 and 222 to output a channel signal 228. The channel signal may be output directly to a loudspeaker, for example a 32-channel loudspeaker, as shown at 230. The signal 228 may be provided to a format conversion circuit 232, which receives as a control input a playback layout signal indicating how the channel signal 228 should be converted. In the embodiment shown in Fig. 2, it is assumed that the conversion is performed in such a way that the signal may be provided to a 5.1 speaker system, as shown at 234. The channel signals 228 may also be provided to a binaural renderer 236, which generates two output signals, such as for headphones, as shown at 238.

[0029] In one embodiment of the present invention, the encoding / decoding system shown in Fig. 1 and Fig. 2 is based on the MPEG-D USAC codec for coding channel signals and object signals (see signals 104 and 106). To increase the efficiency for coding a large amount of objects, MPEG SAOC technology may be used. Three types of renderers may perform the task of rendering objects to channels, rendering channels to headphones, or rendering channels to different loudspeaker setups (see Fig. 2, references 230, 234 and 238). When an object signal is explicitly transmitted or parametrically encoded using SAOC, the corresponding object metadata information 108 is compressed (see signal 126) and multiplexed into the 3D audio bitstream 128. The algorithmic blocks of the overall 3D audio system shown in Figures 1 and 2 are explained in further detail below.

[0030] An optional pre-renderer / mixer 102 may be provided to convert the channel+object input scene into a channel scene before encoding. Functionally, the pre-renderer / mixer 102 is equivalent to the object renderer / mixer described below. Pre-rendering of objects may be desired to ensure a deterministic signal entropy at the encoder input that is essentially independent of the number of simultaneously active object signals. Pre-rendering of objects does not require object metadata transmission. Discrete object signals are rendered into the channel layout that the encoder is configured to use. Object weights for each channel are obtained from the associated object metadata (OAM).

[0031] The USAC encoder 116 is the core codec for loudspeaker channel signals, discrete object signals, object downmix signals and pre-rendered signals. The USAC encoder 116 is based on MPEG-D USAC technology. The USAC encoder 116 handles the coding of the above signals by creating channel and object mapping information based on the geometric and semantic information of the input channel and object assignments. This mapping information describes how the input channels and objects are mapped to USAC channel elements such as channel pair elements (CPEs), single channel elements (SCEs), low frequency effects (LFEs) and quad channel elements (QCEs), and the CPEs, SCEs and LFEs as well as the corresponding information are transmitted to the decoder. All additional payloads such as SAOC data 114, 118 or object metadata 126 are taken into account in the rate control of the encoder. The coding of objects is possible in various ways depending on the rate / distortion and interactivity requirements for the renderer. According to the embodiment, the following object coding variants are possible:

[0032] Pre-rendered Object: The object signal is pre-rendered and mixed into a 22.2 channel signal before encoding. The subsequent coding chain experiences a 22.2 channel signal.

[0033] Discrete Object Waveform: Objects are fed to the encoder as monophonic waveforms. The encoder uses a Single Channel Element (SCE) to transmit the objects in addition to the channel signals. The decoded objects are rendered and mixed at the receiver side. Compressed object metadata information is sent to the receiver / renderer.

[0034] Parametric object waveforms: object properties and their relationships to each other are described by SAOC parameters. The downmix of object signals is coded using USAC. The parameter information is transmitted together. The number of downmix channels is chosen depending on the number of objects and the overall data rate. The compressed object metadata information is transmitted to the SAOC renderer.

[0035] The SAOC encoder 112 and the SAOC decoder 220 for object signals may be based on MPEG SAOC technology. The system is capable of recreating, modifying and rendering some audio objects based on a smaller number of transmitted channels and additional parametric data such as OLD, IOC (inter-object coherence), DMG (downmix gain), etc. The additional parametric data represents a significantly lower data rate than required to transmit all objects individually, which makes the coding extremely efficient. The SAOC encoder 112 takes as input the object / channel signals as monophonic waveforms and outputs the parameter information (packed into the 3D audio bitstream 128) and the SAOC transport channels (encoded and transmitted using single channel elements). The SAOC decoder 220 reconstructs the object / channel signals from the decoded SAOC transport channels 210 and the parameter information 214 and generates the output audio scene based on the playback layout, the restored object metadata information, and optionally based on the user interaction information.

[0036] An object metadata codec (see OAM Encoder 124 and OAM Decoder 224) is provided such that for each object, associated metadata specifying the object's geometric location and volume in 3D space is efficiently coded by quantization of object properties in time and space. The compressed object metadata cOAM 126 is transmitted to the receiver 200 as side information.

[0037] The object renderer 216 utilizes the compressed object metadata to generate object waveforms according to a given playback format. Each object is rendered to an output channel according to its metadata. The output of this block comes from the sum of the partial results. If both channel-based content as well as discrete / parametric objects are decoded, the channel-based waveforms and the rendered object waveforms are mixed by the mixer 226, after which the resulting waveform 228 is output or fed to a post-processor module such as the binaural renderer 236 or the loudspeaker renderer module 232.

[0038] The Binaural Renderer module 236 generates a binaural downmix of the multi-channel audio material such that each input channel is represented by a virtual source. Processing is done frame-by-frame in the QMF (Quadrature Mirror Filter Bank) domain and the binauralization is based on a measured binaural room impulse response.

[0039] The loudspeaker renderer 232 converts between the transmitted channel configuration 228 and the desired playback format. It is sometimes called a "format converter." A format converter performs a conversion to a lower number of output channels, i.e., it produces a downmix.

[0040] 3 shows an example for implementing the format converter 232. The format converter 232, also called loudspeaker renderer, converts between the transmitter channel configuration and the desired playback format. The format converter 232 performs a conversion to a lower number of output channels, i.e. it performs a downmix (DMX) process 240. A downmixer 240, preferably operating in the QMF domain, receives the mixer output signal 228 and outputs the loudspeaker signal 234. A configurator 242, also called controller, may be provided, which receives as control inputs a signal 246 indicating the mixer output layout, i.e. the layout for which the data represented by the mixer output signal 228 is determined, and a signal 248 indicating the desired playback layout. Based on this information, the controller 242 generates, preferably automatically, optimized downmix matrices for a given combination of input and output formats and applies these matrices to the downmixer 240. The format converter 232 allows for standard loudspeaker configurations as well as random configurations with non-standard loudspeaker positions.

[0041] FIG. 4 illustrates an embodiment of the binaural renderer 236 of FIG. 2. The binaural renderer module may provide a binaural downmix of multi-channel audio material. The binauralization may be based on a measured binaural room impulse response. The room impulse response may be considered as a "fingerprint" of the acoustic properties of a real room. The room impulse response may be measured and stored, and any acoustic signal may be provided with this "fingerprint", thereby enabling the simulation of the acoustic properties of the room related to the room impulse response at the listener. The binaural renderer 236 may be configured or programmed to render the output channels into two binaural channels using head-related transfer functions or binaural room impulse responses (BRIRs). For example, in a mobile device, binaural rendering is desired for headphones or loudspeakers attached to such a mobile device. In such a mobile device, constraints may make it necessary to limit the decoder and rendering complexity. In addition to omitting decorrelation in such a processing scenario, it may be preferable to first perform a downmix using the downmixer 250 to an intermediate downmix signal 252, i.e. to a lower number of output channels, resulting in a lower number of input channels for the actual binaural converter 254. For example, 22.2 channel material may be downmixed to a 5.1 intermediate downmix by the downmixer 250, or alternatively, the intermediate downmix may be calculated directly by the SAOC decoder 220 of FIG. 2 in a kind of "shortcut" mode. In that case, binaural rendering only needs to apply 10 HRTF (head-related transfer functions) or BRIR functions to render the five individual channels at different positions, as opposed to applying 44 HRTF or BRIR functions if the 22.2 input channels were to be rendered directly. The convolution operations required for binaural rendering require a lot of processing power, so reducing this processing power while still obtaining an acceptable audio quality is particularly useful for mobile devices.The binaural renderer 236 generates a binaural downmix 238 of the multi-channel audio material 228 such that each input channel (except the LFE channel) is represented by a virtual source. Processing may be done frame-by-frame in the QMF domain. Binauralization is based on measured binaural room impulse responses, and the direct sound and early reflections may be transferred to the audio material via a convolution approach in the pseudo-FFT domain using fast convolution on top of the QMF domain, while the late reverberations may be processed separately.

[0042] Figure 5 shows an example of a room impulse response h(t) 300. The room impulse response contains three components: direct sound 301, early reflections 302 and late reverberation 304. In this way, the room impulse response describes the reflection behavior of a closed reverberant acoustic space when an impulse is emitted. The early reflections 302 are individual reflections with increasing density, and the part of the impulse response where the individual reflections can no longer be distinguished is called the late reverberation 304. The direct sound 301 can be easily identified and separated from the early reflections in the room impulse response, but the transition from the early reflections 302 to the late reverberation 304 is less obvious.

[0043] As explained above, in a binaural renderer, such as the one shown in FIG. 2, various techniques are known for processing a multi-channel audio input signal according to a room impulse response.

[0044] Fig. 6 shows different possibilities for processing an audio input signal with a room impulse response. Fig. 6(a) shows processing the complete audio signal according to the room impulse response, and Fig. 6(b) shows separate processing of the early and late reverberant parts. As shown in Fig. 6(a), an input signal 400, e.g. a multi-channel audio input signal, is received and applied to a processor 402, which is configured or programmed to allow a complete convolution of the multi-channel audio input signal 400 with the room impulse response (see Fig. 5), which in the illustrated embodiment results in a two-channel audio output signal 404. As mentioned above, this approach is considered to be disadvantageous, since using a convolution on the entire impulse response is computationally very costly. Thus, according to another approach, as shown in Fig. 6(b), instead of processing the entire multi-channel audio input signal by applying a full convolution with the room impulse response described with respect to Fig. 6(a), the processing is separated for the early part 301, 302 of the room impulse response 300 (see Fig. 5) and the late reverberant part 302. More specifically, as shown in Fig. 6(b), a multi-channel audio input signal 400 is received, but the signal is applied in parallel to a first processor 406 for processing the early part, i.e. for processing the audio signal according to the direct sound 301 and the early reflections 302 in the room impulse response 300 shown in Fig. 5. The multi-channel audio input signal 400 is also applied to a processor 408 for processing the audio signal according to the late reverberation 304 of the room impulse response 300. In the embodiment shown in Fig. 6(b), the multi-channel audio input signal may also be applied to a downmixer 410 for downmixing the multi-channel signal 400 to a signal having a lower number of channels. The output of the downmixer 410 is then applied to the processor 408. The outputs of the processors 406 and 408 are combined at 412 to produce a two-channel audio output signal 404'.

[0045] In a binaural renderer, as mentioned above, it may be desirable to process the direct sound and early reflections separately from the late reverberation, mainly for the purpose of reducing computational complexity. The direct sound and early reflection processing may be transferred to the audio signal, for example, by a convolution technique performed by the processor 406 (see FIG. 6(b)), while the late reverberation may be replaced by a synthetic reverberation performed by the processor 408. The overall binaural output signal 404' is then a combination of the convolution result provided by the processor 406 and the synthetic reverberation signal provided by the processor 408.

[0046] This process is also described in the prior art document [1]. The results of the above described technique should be as perceptually equivalent as possible to the results of the full impulse response convolution, the full transformation technique described with respect to FIG. 6(a). However, when an audio signal, or more generally, audio material, is convolved with the direct sound and the early reflection parts of the impulse response, the result is that the different channels are summed to form an overall sound signal associated with the playback signal to one ear of the listener. The reverberation is not, however, calculated from this overall signal, but is generally a reverberation signal of one channel or a downmix of the original input audio signal. It has been determined by the inventors of the present invention that the late reverberation is therefore not adequately matched to the convolution result provided by the processor 406. It has been found that the appropriate level of reverberation depends on both the input audio signal and the room impulse response 300. The influence of the impulse response is achieved by using the reverberation characteristics as input parameters of a reverberator that may be part of the processor 408, these input parameters being obtained from the analysis of the measured impulse responses, e.g. frequency-dependent reverberation times and frequency-dependent energy measures. These measures can generally be determined from a single impulse response, for example by calculating the energy and RT60 reverberation time in an octave filter bank analysis, or are averages of the results of multiple impulse response analyses.

[0047] However, despite these input parameters provided to the reverberator, it has been found that when using a synthetic reverberation approach such as that described with respect to FIG. 6(b), the contribution of the input audio signal to the reverberation is not fully preserved. For example, due to the downmix used to generate the synthetic reverberation tail, the contribution of the input audio signal is lost. The resulting level of reverberation is therefore not perceptually equivalent to the results of a full convolution approach, especially when the input signal contains multiple channels.

[0048] So far, there is no known method to compare the amount of late reverberation with the results of a full convolution method or to match it to the convolution result. There are several techniques that attempt to rate the quality of the late reverberation or how natural it sounds. For example, in one method, a loudness measure for natural acoustic reverberation is defined, which predicts the perceived loudness of the reverberation using a loudness model. This method is described in the prior art [2], and the level can be adapted to a target value. The drawback of this method is that it relies on a model of human hearing, which is complex and inaccurate. It also requires a target loudness to provide a scaling factor for the late reverberation, which can be found using the full convolution result.

[0049] Another method described in prior art document [3] uses a cross-correlation criterion for artificial reverberation quality testing, but it is only applicable for testing different reverberation algorithms, but not for multi-channel audio, not for binaural audio and not for qualifying the scaling of late reverberation.

[0050] Another possible approach is to use the number of input channels at the considered ear as a scaling factor, but this does not give a perceptually correct scaling, since the perceived amplitude of the overall sound signal depends on the correlation of the different audio channels and not only on the number of channels.

[0051] Thus, according to the inventive approach, a signal-dependent scaling method is provided that adapts the level of reverberation according to the input audio signal. As mentioned above, it is desired that the perceived level of reverberation matches the level of reverberation when using a full convolution approach for binaural rendering, and therefore the determination of a measure for an appropriate level of reverberation is important to achieve good sound quality. According to an embodiment, the audio signal is processed separately with an early part of the room impulse response and a late reverberation, and processing the late reverberation comprises generating a scaled reverberation signal, the scaling being dependent on the audio signal. The processed early part of the audio signal and the scaled reverberation signal are combined into an output signal. According to one embodiment, the scaling is dependent on the state of one or more input channels of the audio signal (e.g., the number of input channels, the number of active input channels and / or the activity in the input channels). According to another embodiment, the scaling is dependent on a predefined or calculated correlation measure for the audio signal. Alternative embodiments may perform the scaling based on a combination of the state of one or more input channels and a predefined or calculated correlation measure.

[0052] According to an embodiment, the scaled reverberation signal may be generated by applying a gain factor determined based on the state of one or more input channels of the audio signal, or based on a predefined or calculated correlation measure for the audio signal, or based on a combination thereof.

[0053] According to an embodiment, processing the audio signal separately comprises processing the audio signal with early reflection parts 301, 302 of the room impulse response 300 during a first process and processing the audio signal with diffuse reverberation 304 of the room impulse response 300 during a second process, which is different and separate from the first process. The change from the first process to the second process is performed at a transition time. According to a further embodiment, in the second process the diffuse (late) reverberation 304 can be replaced with a synthetic reverberation. In this case the room impulse response applied to the first process contains only the early reflection parts 300, 302 (see FIG. 5) and not the late diffuse reverberation 304.

[0054] In the following, an embodiment of the inventive approach is described in more detail, in which the gain factors are calculated accordingly based on a correlation analysis of the input audio signals. Figure 7 shows a block diagram of a signal processing unit, such as a binaural renderer, operating in accordance with the teachings of the present invention. The binaural renderer 500 comprises a first branch including a processor 502 that receives from an input 504 an audio signal x[k] comprising N channels. When the processor 502 is part of the binaural renderer, it processes the input signal 504 to produce an output signal 506 x conv[k]. More specifically, the processor 502 causes a convolution of the audio input signal 504 with the direct sound and early reflections of the room impulse response, which may be provided to the processor 502 from an external database 508 holding a plurality of recorded binaural room impulse responses. The processor 502 may operate based on the binaural room impulse response provided by the database 508, as described above, thereby generating an output signal 502 having only two channels. The output signal 506 is provided from the processor 502 to a summer 510. The input signal 504 is further provided to a reverberation branch 512, which includes a reverberator processor 514 and a downmixer 516. The downmixed input signal is provided to the reverberator 514, which generates a reverberation signal r[k] at the output of the reverberator 514, which may include only two channels, based on reverberator parameters such as reverberation RT60 and reverberation energy held in databases 518 and 520, respectively. The parameters stored in the databases 518 and 520 may be obtained from the stored binaural room impulse responses by a suitable analysis 522, as indicated by the dashed line in FIG.

[0055] The reverberation branch 512 further includes a correlation analysis processor 524 which receives the input signal 504 and generates at its output a gain factor g. Further, a gain stage 526 is provided coupled between the reverberator 514 and the summer 510. The gain stage 526 is controlled by the gain factor g, such that at the output of the gain stage 526 a scaled reverberation signal r g [k] and generate the reverberation signal r gThe gain factor y[k] is applied to a summer 510, which combines the early processing portion and the reverberation signal to provide an output signal y[k] that also includes two channels. Optionally, the reverberation branch 512 may comprise a low pass filter 528 coupled between the processor 524 and the gain stage to smooth the gain factor over several audio frames. Optionally, a delay element 530 may also be provided between the output of the gain stage 526 and the summer 510 to delay the scaled reverberation signal so that it matches the transition between the early reflections and the reverberation in the room impulse response.

[0056] As explained above, Fig. 7 is a block diagram of a binaural renderer that processes direct sound and early reflections separately from late reverberation. As can be seen, the input signal x[k] processed with the direct and early reflections of the binaural room impulse response is the signal x conv This signal is then divided into the reverberation signal components r[k] as shown. g The scaled or weighted reverberation component is then forwarded to an adder 510 for addition to the input signal x[k]. This signal is generated by feeding a downmix of the input signal x[k], for example a stereo downmix, to a reverberator 514, followed by a multiplier or gain stage 526 receiving the downmix reverberation signal r[k] and a gain factor g. The gain factor g is obtained by correlation analysis of the input signal x[k] performed by the processor 524, and may be smoothed over time by a low pass filter 528, as described above. The scaled or weighted reverberation component may optionally be delayed by a delay element 530 so that its onset coincides with the transition point from early reflections to late reverberation, and thus an output signal y[k] is obtained at the output of the adder 510.

[0057] The multi-channel binaural renderer shown in Fig. 7 introduces a synthetic two-channel late reverberation, and to overcome the above-described shortcomings of the conventional approach, according to the inventive approach, the synthetic late reverberation is scaled by a gain factor g to match the perception to the results of a full convolution approach. The superposition of multiple channels at the listener's ear (e.g. up to 22.2) is correlation-dependent. For that reason, the late reverberation may be scaled according to the correlation of the input signal channels, and an embodiment of the inventive approach provides a correlation-based time-dependent scaling method to determine the appropriate amplitude of the late reverberation.

[0058] To calculate the scaling factors, a correlation measure based on correlation coefficients is introduced and according to an embodiment is defined in a two-dimensional time-frequency domain, e.g., the QMF domain. A correlation value between -1 and 1 is calculated for each multidimensional audio frame, where each audio frame is defined by a number of frequency bands N, a number of time slots per frame M, and a number of audio channels A. One scaling factor is obtained per ear per frame.

[0059] In the following, one embodiment of the inventive technique is described in more detail. Firstly, reference is made to the correlation measure used in the correlation analysis processor 524 of Fig. 7. The correlation measure is, according to this embodiment, based on the Pearson product moment coefficient (also known as the correlation coefficient), which is calculated by dividing the covariance of two variables X, Y by the product of their standard deviations as follows:

number

number

number

number

[0060] According to the embodiments described above, the scaling was determined based on a calculated correlation measure for the audio signals. This is advantageous, despite the additional computational resources required, for example when it is desired to obtain a correlation measure for each currently processed audio signal individually.

[0061] However, the invention is not limited to such an approach. According to other embodiments, rather than calculating the correlation measure, a predefined correlation measure may also be used. Using a predefined correlation measure is advantageous as it reduces the computational complexity in the process. The predefined correlation measure may have a fixed value, for example between 0.1 and 0.9, which may be empirically determined based on an analysis of multiple audio signals. In such a case, the correlation analysis 524 may be omitted and the gain of the gain stage may be set by an appropriate control signal.

[0062] According to other embodiments, the scaling may depend on the state of one or more input channels of the audio signal (e.g., the number of input channels, the number of active input channels and / or the activity in the input channels). This is advantageous since the scaling can be easily determined from the input audio signal with reduced computational overhead. For example, the scaling may be determined by simply determining the number of channels in the original audio signal to be downmixed to the currently considered downmix channel, which includes a reduced number of channels compared to the original audio signal. Alternatively, the number of active channels (channels showing some activity in the current audio frame) to be downmixed to the currently considered downmix channel may form the basis for scaling the reverberant signal. This may be done in block 524.

[0063] In the following, an embodiment is described in detail which determines the scaling of the reverberant signal based on the state of one or more input channels of the audio signal and based on a correlation measure (either fixed or calculated as explained above). According to such an embodiment, a gain factor or scaling factor g is defined as follows:

number

number

number

number

[0064] When considering for example a 5.1 input signal, the following applies: Channels 1, 3, and 4 are downmixed to downmix channel 1 (see matrix Q above), In frame n, The active channels are channels 1, 2, and 4. K in is the number of channels at the intersection {1,4}, K in (n)=2 In frame n+1, The active channels are channels 1, 2, 3, and 4. K in is the number of channels at the intersection {1,3,4}, K in (n+1)=3. An audio channel (in a predefined frame) may be considered active if it has an amplitude or energy within the predefined frame that exceeds a preset threshold, for example, according to an embodiment, activity in an audio channel (in a predefined frame) may be defined as follows: The sum or maximum of the absolute amplitudes of the signal (in the time domain, QMF domain, etc.) during the frame is greater than 0, or ·The sum or maximum value of the signal energy (the absolute squared value of the amplitude in the time domain or QMF domain) during the frame is greater than 0. Instead of 0, another threshold (for maximum energy or amplitude) greater than 0 may also be used, for example a threshold of 0.01.

[0065] According to an embodiment, the number of active channels (time-varying) or the fixed number K of included channels (downmix matrices not equal to 0) in the downmix channel in A gain factor for each ear is provided that depends on . The factors are assumed to increase linearly between the globally uncorrelated and globally correlated cases. Globally uncorrelated means no inter-channel dependency (correlation value is 0), whereas globally correlated means the signals are weighted versions of each other (with an offset phase difference, correlation value is 1).

[0066] As mentioned above, the gain or scaling factor g may be smoothed across the audio frame by a low pass filter 528. The low pass filter 528 calculates g for a frame size k as follows: s t yields a smoothed gain factor of (t) s may have a time constant of

number

number

number

[0067] Fig. 8 shows a schematic diagram of a binaural renderer 800 for binaural processing of an audio signal according to an embodiment of the present invention. Fig. 8 also provides an overview of the QMF domain processing in the binaural renderer. At the input 802, the binaural renderer 800 receives the audio signal to be processed, for example an input signal comprising N channels and 64 QMF bands. Furthermore, the binaural renderer 800 receives several input parameters for controlling the processing of the audio signal. The input parameters are a Binaural Room Impulse Response (BRIR) 804 for 2xN channels and 64 QMF bands, an indication K of the maximum band to be used for the convolution of the audio input signal with the early reflections part of the BRIR 804, max 806, and the reverberator parameters 808 and 810 (RT60 and reverberation energy) mentioned above. The binaural renderer 800 comprises a fast convolution processor 812 for processing the input audio signal 802 with an initial part of the received BRIR 804. The processor 812 outputs at its output two channels and K max8 and 64 QMF bands. Besides the initial processing branch with the fast convolution processor 812, the binaural renderer 800 also comprises a reverberation branch with two reverberators 816a and 816b, each of which receives the RT60 information 808 and the reverberation energy information 810 as input parameters. The reverberation branch further comprises a stereo downmix processor 818 and a correlation analysis processor 820, both of which also receive the input audio signal 802. Furthermore, two gain stages 821a and 821b are provided between the stereo downmix processor 818 and the respective reverberators 816a and 816b to control the gain of the downmix signal 822 provided by the stereo downmix processor 818. The stereo downmix processor 818 provides a downmix signal 822 having two bands and 64 QMF bands based on the input signal 802. The gains of the gain stages 821a and 821b are controlled by respective control signals 824a and 824b provided by the correlation analysis processor 820. The gain controlled downmix signals are input to respective reverberators 816a and 816b to generate respective reverberation signals 826a, 826b. The early processing signal 814 and the reverberation signals 826a, 826b are received by a mixer 828, which combines the received signals into an output audio signal 830 having two channels and 64 QMF bands. Furthermore, according to the invention, the fast convolution processor 812 and the reverberators 816a and 816b receive an additional input parameter 832 indicative of the transition in the room impulse response 804 from the early part to the late reverberation, determined as explained above.

[0068] A binaural renderer module 800 (e.g., binaural renderer 236 of FIG. 2 or FIG. 4) has as input 802 the decoded data stream. The signal is processed by a QMF analysis filter bank as outlined in ISO / IEC 14496-3:2009, section 4.B.18.2, with modifications described in ISO / IEC 14496-3:2009, section 8.6.4.2. The renderer module 800 may also process QMF domain input data, in which case the analysis filter bank may be omitted. A binaural room impulse response (BRIR) 804 is represented as a complex QMF domain filter. The conversion from a time domain binaural room impulse response to a complex QMF filter representation is outlined in ISO / IEC FDIS 23003-1:2006, Annex B. In the complex QMF domain, the BRIR 804 is limited to a certain number of time slots so that it includes only the early reflections 301, 302 (see FIG. 5) and does not include the late diffuse reverberation 304. The transition point 832 from early reflections to late reverberation is determined by analysis of the BRIR 804, for example in a pre-processing step of binaural processing, as explained above. The QMF domain audio signal 802 and the QMF domain BRIR 804 are then processed by band-wise fast convolution 812 to perform binaural processing. QMF domain reverberators 816a, 816b are used to generate two-channel QMF domain late reverberation 826a, 826b. The reverberation modules 816a, 816b use a set of frequency-dependent reverberation times 808 and energy values ​​810 to adapt the characteristics of the reverberation. The reverberation waveform is based on a stereo downmix 818 of the audio input signal 802, which is adaptively scaled 821a, 821b in amplitude according to a correlation analysis 820 of the multi-channel audio signal 802. The two-channel QMF domain convolution result 814 and the two-channel QMF domain reverberation 816a, 816b are then combined 828 and finally two QMF synthesis filter banks compute a binaural time domain output signal 830 as outlined in ISO / IEC 14496-3:2009, clause 4.6.18.4.2. The renderer can also generate QMF domain output data.In that case the synthesis filter bank is omitted.

[0069] definition The audio signal 802 fed into the binaural renderer module 800 is referred to below as the input signal. The audio signal 830 resulting from the binaural processing is referred to as the output signal. The input signal 802 of the binaural renderer module 800 is the audio output signal of the core decoder (see for example signal 228 in FIG. 2). The following variable definitions are used: [Table 1]

[0070] process Now, regarding the processing of the input signal, the Binaural Renderer module operates on consecutive non-overlapping frames of length = 2048 time-domain samples of the input audio signal and outputs one frame of samples for each processed input frame of length.

[0071] (1) Initialization and preprocessing The initialization of the binaural processing block takes place before any processing of the audio samples provided by the core decoder (see for example decoder 200 in FIG. 2) occurs. The initialization consists of several processing steps.

[0072] (a) Analytical value reading The reverberator modules 816a, 816b take as input parameters a frequency-dependent set of reverberation times 808 and energy values ​​810. These values ​​are read from the interface at the initialization of the binaural processing module 800. Additionally, the transition time 832 from early reflections to late reverberation in time domain samples is read. The values ​​may be stored in a binary file written with 32 bits per sample, float values, little endian ordering. The read values ​​required for processing are set out in the table below. [Table 2]

[0073] (b) BRIR reading and preprocessing The binaural room impulse responses 804 are read from two dedicated files that store the left and right ear BRIRs respectively. The time domain samples of the BRIRs are stored in integer wave files with a resolution of 24 bits per sample and 32 channels. The ordering of the BRIRs in the files is as described in the table below. [Table 3]

[0074] If there is no BRIR measured at one of the loudspeaker positions, the corresponding channel in the wave file contains a value of 0. The LFE channel is not used for binaural processing.

[0075] As a pre-processing step, a given set of Binaural Room Impulse Responses (BRIRs) are transformed from time-domain filters to complex-valued QMF domain filters. The implementation of the given time-domain filters in the complex-valued QMF domain is done according to ISO / IEC FDIS 23003-1:2006, Annex B. The prototype filter coefficients for the filter transformation are used according to ISO / IEC FDIS 23003-1:2006, Annex B, Table B.1. 1 ≤ v ≤ L trans,n The complex-valued QMF domain filter To obtain TIFF2025039612000022.tif26119, 1≦v≦L trans The time domain representation is TIFF2025039612000023.tif25121 is processed. (2) Audio signal processing The audio processing block of the binaural renderer module 800 receives the N in Obtain time domain audio samples 802 for N input channels, out= A binaural output signal 830 consisting of two channels is generated. The process takes as input the following: · decoded audio data 802 from the core decoder; A complex QMF domain representation of the early reflections part of the BRIR set 804, and · Frequency-dependent parameter sets 808, 810, 832 used by the QMF domain reverberators 816a, 816b to generate late reverberations 826a, 826b.

[0076] (a) QMF analysis of an audio signal As the first processing step, the binaural renderer module processes the N in Channel time domain = 2048 time domain samples of the input signal TIFF2025039612000024.tif24132, dimension L n N = 32 QMF time slots (slot index n) and K = 64 frequency bands (band index k). in Transform the channel into a QMF domain signal representation 802 . The QMF analysis outlined in ISO / IEC 14496-3:2009, section 4.B.18.2 and modified in ISO / IEC 14496-3:2009, section 8.6.4.2, is used to analyze the time-domain signal. This is performed on frames of TIFF2025039612000025.tif1921, where 1≦v≦L and 1≦n≦L n QMF domain signal Frame TIFF2025039612000026.tif24136 is acquired.

[0077] (b) Fast convolution of QMF domain audio signal with QMF domain BRIR Next, band-wise fast convolution 812 is performed to process the QMF domain audio signal 802 and the QMF domain BRIR 804. An FFT analysis may be performed on each QMF frequency band for each channel of the input signal 802 and each BRIR 804. Due to the complex values ​​in the QMF domain, one FFT analysis is performed on the real part of the QMF domain signal representation and one FFT analysis is performed on the imaginary part of the QMF domain signal representation. The results are then combined to form the final band-wise complex-valued pseudo-FFT domain signal as follows: TIFF2025039612000027.tif13166 and the band-wise complex-valued BRIR is formed as follows, In the left ear TIFF2025039612000028.tif13150 Right ear TIFF2025039612000029.tif13152. The length of the FFT transform is the length L of the complex-valued QMF domain BRIR filter. trans,n and QMF region time slot L n and thus, L FFT =L trans,n +L n -1. The complex-valued pseudo-FFT domain signal is then multiplied by a complex-valued pseudo-FFT domain BRIR filter to form a fast convolution result. A vector m is used to signal which channel of the input signal corresponds to which BRIR pair in the BRIR data set. conv is used. This multiplication is 1≦k≦K. max The QMF frequency band k is then calculated for all QMF frequency bands k where k is the maximum band K max is determined by the QMF band, which represents either 18 kHz or the maximum signal frequency present in the audio signal from the core decoder. f max =min(f max,decoder ,18kHz). The multiplication results from each audio input channel with each BRIR pair are 1≦k≦K max are summed over each QMF frequency band, so that the intermediate two channels K max A band-pass pseudo-FFT domain signal is produced. TIFF2025039612000030.tif30136 and TIFF2025039612000031.tif29137 is the QMF domain frequency band k Pseudo FFT convolution result in The file is TIFF2025039612000032.tif19139. Then, a band-wise FFT synthesis is performed and the convolution result is transformed back to the QMF domain, so that 1≦n≦L FFT and 1≦k≦K max L FFT Time slot Intermediate 2 channel K with TIFF2025039612000033.tif19139 max A band QMF domain signal results. For each QMF domain input signal frame with L = 32 time slots, a convolution result signal frame with L = 32 time slots is returned. FFT - 32 time slots are stored and overlap-add processing is performed in the following frame.

[0078] (c) Late reverberation generation As the second intermediate signals 826a, 826b, A reverberant signal called TIFF2025039612000034.tif20132 is generated by the frequency domain reverberator modules 816a, 816b. The frequency domain reverberators 816a, 816b take as input: QMF domain stereo downmix of one frame of the input signal 822, A parameter set including a frequency dependent reverberation time 808 and an energy value 810. The frequency domain reverberators 816a, 816b return the two-channel QMF domain late reverberation tail. The maximum number of usable bands of the frequency-dependent parameter set is calculated according to the maximum frequency.

[0079] First, the input signal A QMF domain stereo downmix 818 of one frame of TIFF2025039612000035.tif2227 is performed to form the input of the reverberator by a weighted sum of the input signal channels. The weighting gains are calculated using the downmix matrix M DMX The weighting gains are real-valued and non-negative, and the downmix matrix has dimension N out ×N in It contains the non-zero values ​​that cause the channels of the input signal to be mapped to one of the two output channels.

[0080] Channels representing loudspeaker positions on the left hemisphere are mapped to the left output channel and channels representing loudspeakers on the right hemisphere are mapped to the right output channel. The signals of these channels are weighted by a factor of 1. Channels representing loudspeakers in the median plane are mapped to both output channels of the binaural signal. The input signals of these channels are weighted by the following factors: TIFF2025039612000036.tif39115

[0081] Furthermore, an energy equalization step is performed in the downmix, which adapts the band-wise energy of one downmix channel to be equal to the sum of the band-wise energies of the input signal channels contained in this downmix channel. This energy equalization is performed by band-wise multiplication with the following real-valued coefficients: TIFF2025039612000037.tif44124

[0082] Factor C eq,k is restricted to the interval [0.5,2]. A numerical constant ε is introduced to avoid division by zero. The downmix is ​​also max The values ​​in all higher frequency bands are set to zero.

[0083] FIG. 9 illustrates a schematic representation of the processing in the frequency domain reverberators 816a, 816b of a binaural renderer 800 according to one embodiment of the present invention.

[0084] In the frequency domain reverberator, a mono downmix of the stereo input is computed using an input mixer 900. This is done non-coherently by applying a 90° phase shift on the second input channel.

[0085] This mono signal is then fed into a feedback delay loop 902 in each frequency band k, which creates a decaying sequence of impulses. This is followed by a parallel FIR decorrelator that distributes the signal energy in a decaying manner during the intervals between the impulses, creating decoherence between the output channels. A decaying filter tap density is applied to create the energy decay. The filter tap phase operation is limited to four options to implement a sparse, multiplier-less decorrelator.

[0086] After the calculation of the reverberation, an inter-channel coherence (ICC) correction 904 is included in the reverberator module for each QMF frequency band. In the ICC correction step, a frequency-dependent direct gain g direct and the cross-mix gain g cross and are used.

[0087] The amount of energy and reverberation time for different frequency bands are contained in the input parameter set. Values ​​are given at a number of frequency points that are internally mapped to the K=64 QMF frequency bands.

[0088] Final intermediate signal Two instances of the frequency domain reverberator are used to compute TIFF2025039612000038.tif19124. The TIFF2025039612000039.tif2647 signal is the first output channel of the first instance of the reverberator. TIFF2025039612000040.tif2549 is the second output channel of the second instance of the reverberator. They are combined into the final reverberant signal frame with dimensions 2 channels, 64 bands and 32 time slots. To ensure correct scaling of the reverberator output, the stereo downmix 822 is scaled 821a,b in both time periods according to the correlation measure 820 of the input signal frames. The scaling factor is the correlation coefficient c between 0 and 1. corr Linearly depending on It is defined as a value in the interval TIFF2025039612000041.tif32157, where TIFF2025039612000042.tif30137 and TIFF2025039612000043.tif45119

[0089] however, TIFF2025039612000044.tif2237 is channel A over one time slot n, the operator {*} denotes the complex conjugate, TIFF2025039612000045.tif2310 is the QMF domain signal in the actual signal frame This is a zero-mean version of TIFF2025039612000046.tif2112.

[0090] c corr is calculated twice: once for the channels A, B that are active in the actual signal frame F and that are included in the left channel of the stereo downmix, and once for the channels A, B that are active in the actual signal frame F and that are included in the right channel of the stereo downmix. DMX,act is one downmix channel A (not equal to 0 in the downmix matrix M DMX is the number of input channels that are downmixed to the Ath row of the current frame (the number of matrix elements in the Ath row of

[0091] The scaling factor is then: TIFF2025039612000047.tif23170 The scaling factor is smoothed over the audio signal frames by a first-order low-pass filter, so that the smoothed scaling factor The result is TIFF2025039612000048.tif25154.

[0092] The scaling factors are initialized in the first audio input data frame by time domain correlation analysis using the same procedure.

[0093] The input of the first reverberator instance is the scaling factor TIFF2025039612000049.tif2748 and the input of the second reverberator instance is scaled by the scaling factor Scaled with TIFF2025039612000050.tif2751.

[0094] (d) Synthesis of the convolution result and late reverberation Next, for one QMF domain audio input frame, the convolution result 814, TIFF2025039612000051.tif18132, and reverberator outputs 826a, 826b, TIFF2025039612000052.tif20130 is then combined by a mixing process 828 that sums the two signals band-wise. max Since this is only done in the band up to K max The upper band, which is higher than Note that TIFF2025039612000053.tif2345 is 0. The late reverberation output is d=((L trans It is delayed by an amount of (-20·64+1) / 64+0.5)+1 time slots. The delay d takes into account the transition time from early to late reflections in the BRIR, and the initial delay of the reverberator of 20 QMF time slots, as well as the analysis delay of 0.5 QMF time slots for the QMF analysis of the BRIR, to ensure the insertion of late reverberation in the appropriate time slot. The composite signal at TIFF2025039612000054.tif2629 is Calculated by TIFF2025039612000055.tif24105.

[0095] (e) QMF synthesis of binaural QMF domain signals QMF area output signal One two-channel frame of 32 time slots of TIFF2025039612000056.tif2428 is converted to a two-channel time-domain signal frame with length L by QMF synthesis according to ISO / IEC 14496-3:2009, clause 4.6.18.4.2., thereby producing a final time-domain output signal 830, The result is TIFF2025039612000057.tif23114.

[0096] According to the inventive approach, the synthetic or artificial late reverberation is scaled taking into account the characteristics of the input signal, thereby improving the quality of the output signal while taking advantage of the reduced computational complexity obtained by separate processing, and as can be seen from the above, no additional hearing model or target reverberation loudness is required.

[0097] It should be noted that the present invention is not limited to the embodiments described above. For example, it should be noted that although the above embodiments are described in the context of the QMF domain, other time-frequency domains, such as the STFT domain, may also be used. Also, the scaling factor may be calculated in a frequency-dependent manner such that the correlation is not calculated over the entire number of frequency bands, i.e., i∀[1,N], but in some S subsets defined as follows: i1∀[1,N1],i2∀[N1+1,N2],...,i S ∀[N S-1 +N]

[0098] Also, smoothing can be applied across frequency bands, or bands can be synthesized according to certain rules, for example according to the frequency resolution of hearing. The smoothing can be adapted to different time constants, for example according to frame size or listener preferences.

[0099] The techniques of the present invention can also be applied for various frame sizes, even a frame size of just one time slot in the time-frequency domain is possible.

[0100] According to an embodiment, different downmix matrices may be used for the downmix, for example symmetric downmix matrices or asymmetric matrices.

[0101] The correlation measure may be derived from parameters transmitted in the audio bitstream, for example the inter-channel coherence in MPEG Surround or SAOC. Also, according to an embodiment, it is possible to exclude some values ​​of the matrix from the average calculation, if necessary, for example erroneously calculated values ​​or values ​​on the main diagonal, autocorrelation values.

[0102] The process can be done in the encoder, for example when applying a low-complexity binaural profile, instead of using it in a binaural renderer at the decoder side, which results in some representation of the scaling factor, such as the scaling factor itself, a correlation measure between 0 and 1, etc., and these parameters are transmitted in the bitstream from the encoder to the decoder for a fixed downstream matrix.

[0103] Also, note that while the embodiments described above describe applying gain following the reverberator 514, according to other embodiments gain may also be applied prior to the reverberator 514 or within the reverberator, for example by modifying the gain within the reverberator 514. This is advantageous as less computation may be required.

[0104] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or devices corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, any one or more of the most important method steps may be performed by such an apparatus.

[0105] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium such as a digital storage medium, e.g., a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, and EPROM, EEPROM or FLASH memory, having electronically readable control signals stored thereon that cooperate (or are capable of cooperating) with a programmable computer system such that the respective methods are performed. Thus, the digital storage medium may be computer readable.

[0106] Some embodiments according to the invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0107] Generally, embodiments of the present invention may be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0108] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0109] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0110] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) comprising recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0111] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be adapted to be transferred via a data communication connection, for example via the Internet.

[0112] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or programmed to perform one of the methods described herein.

[0113] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0114] Further embodiments according to the invention comprise an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may comprise, for example, a file server for transferring the computer program to the receiver.

[0115] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware apparatus.

[0116] The above-described embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the configurations and details described herein are obvious to others skilled in the art. Therefore, it is intended not to be limited by the specific details presented as descriptions and explanations of the embodiments herein, but only by the scope of the claims that follow.

[0117] literature [1] Expanded in M.R. Schroeder, "Digital Simulation of Sound Transmission in Reverberant Spaces", The Journal of the Acoustical Society of America, VoS. 47, pp. 424-431 (1970), and J.A. Moorer, "About This Reverberation Business", Computer Music Journal, Vol. 3, no. 2, pp. 13-28, MIT Press (1979). [2] Uhle, Christian, Paulus, Jouni, and Herre, Jurgen: "Predicting the Perceived Level of Late Reverberation Using Computational Models of Loudness" Proceedings, 17th International Conference on Digital Signal Processing (DSP), July 6-8, 2011, Corfu, Greece. [3] Czyzewski, Andrzej: “A Method of Artificial Reverberation Quality Testing” J.Audio Eng.Soc.,Vol.38,No 3,1990.

Claims

1. A method for processing an audio signal (504, 802) according to a room impulse response (300), the method comprising: - separately processing (502, 514, 812, 816a, 816b) the audio signal (504, 802) using an early part (301, 302) of the room impulse response (300) and a late reverberation (304), wherein processing the late reverberation (304) comprises generating a scaled reverberation signal, the scaling being dependent on the audio signal (504, 802); synthesizing the audio signal processed with the initial portion of the room impulse response and the scaled reverberation signal; A method comprising:

2. The method of claim 1 , wherein the scaling is dependent on a state of one or more input channels of the audio signal (504, 802).

3. 3. The method of claim 2, wherein the state of the one or more input channels of the audio signal (504, 802) includes one or more of a number of input channels, a number of active input channels, and activity in the input channels.

4. The method of any one of claims 1 to 3, wherein the scaling depends on a predefined or calculated correlation measure of the audio signal (504, 802).

5. 5. The method of claim 1, wherein generating the scaled reverberation signal comprises applying a gain factor, the gain factor being determined based on the state of the one or more input channels of the audio signal and / or based on the predefined or calculated correlation measure for the audio signal.

6. 6. The method of claim 5, wherein generating the scaled reverberant signal comprises applying the gain factor before, during or after processing the late reverberation of the audio signal.

7. The gain factor is determined as follows: g=c ｕ +ρ・(c ｃ -c ｕ ) however ρ = a predefined or calculated correlation measure for the audio signal (504, 802); c ｕ , c ｃ = a factor indicative of the state of the one or more input channels of the audio signal (504, 802), ｕ refers to a totally uncorrelated channel, and c ｃ relates to the overall correlated channels, The method according to claim 5 or 6.

8. c ｕ and c ｃ is determined as follows: [0010] [0025] however K ｉｎ = the number of active or fixed downmix channels, The method of claim 7.

9. The method of any one of claims 5 to 8, wherein the gain factor is low-pass filtered over multiple audio frames.

10. The gain factor is low pass filtered as follows: [0030] [0045] [0050] however t ｓ = time constant of the low pass filter t ｉ = frame t ｉ Audio Frame in g ｓ = smoothed gain factor k = frame size, and f ｓ = sampling frequency, 10. The method of claim 9.

11. The method of claim 1 , wherein generating the scaled reverberation signal comprises a correlation analysis of the audio signal (504, 802).

12. 12. The method of claim 11, wherein the correlation analysis of the audio signal (504, 802) includes determining a composite correlation measure for audio frames of the audio signal (504, 802), the composite correlation measure being calculated by combining correlation coefficients for multiple channel combinations of an audio frame, each audio frame including one or more time slots.

13. The method of claim 12 , wherein combining the correlation coefficients comprises averaging multiple correlation coefficients for the audio frames.

14. Determining the composite correlation measure comprises: (i) calculating a global average value for each channel of the one audio frame; (ii) calculating a zero-mean audio frame by subtracting the mean value from the corresponding channel; (iii) calculating the correlation coefficient for a plurality of channel combinations; and (iv) calculating the composite correlation measure as the average of the plurality of correlation coefficients; The method of claim 11 or 12, comprising:

15. The correlation coefficient for the channel combination is calculated as follows: [006] however ρ[m,n] = correlation coefficient, σ(x ｍ [j]) = standard deviation over one time slot j of channel m, σ(x ｎ [j]) = standard deviation over one time slot j of channel n, x ｍ , x ｎ = zero-mean variable, i∀[1,N] = frequency band, j∀[1,M] = time slot, m, n ∀[1,K] = channel, * = complex conjugate, 15. The method according to any one of claims 11 to 14.

16. 16. The method of claim 1, comprising delaying the scaled reverberation signal to coincide with a transition point in the room impulse response (300) from early reflections to late reverberation (304).

17. 17. The method of claim 1, wherein processing the late reverberation (304) of the audio signal (504, 802) comprises downmixing the audio signal (504, 802) and applying the downmixed audio signal to a reverberator.

18. A non-tangible computer product comprising a computer readable medium having stored thereon instructions for performing the method of any one of claims 1 to 17 when executed by a computer.

19. an input for receiving an audio signal (504, 802); an early part processor for processing the received audio signal (504, 802) according to an early part (301, 302) of a room impulse response (300); a late reverberation processor for processing the received audio signal (504, 802) according to late reverberation of the room impulse response (300), the late reverberation processor being configured to generate a scaled reverberation signal, the scaling being dependent on the received audio signal (504, 802); an output for combining the processed initial portion of the received audio signal (504, 802) and the scaled reverberation signal into an output audio signal; A signal processing unit comprising:

20. The late reverberation processor comprises: a reverberator receiving the audio signal (504, 802) and generating a reverberation signal; a gain stage coupled to an input or an output of the reverberator and controlled by a gain factor; 20. The signal processing unit of claim 19, comprising:

21. 21. A signal processing unit according to claim 20, comprising a correlation analyzer for generating said gain factor dependent on said audio signal (504, 802).

22. a low pass filter coupled to the gain stage; a delay element coupled between the gain stage and a summer; 22. The signal processing unit of claim 20 or 21, further comprising at least one of:

23. A binaural renderer comprising a signal processing unit according to any one of claims 19 to 22.

24. A signal processing unit according to any one of claims 19 to 22 or a binaural renderer according to claim 23 for processing an audio signal prior to coding.

23. An audio encoder for coding the audio signal, comprising:

25. A signal processing unit according to any one of claims 19 to 22 or a binaural renderer according to claim 23 for processing the decoded audio signal.

1. An audio decoder for decoding an encoded audio signal, comprising:

Images