Mode reverberation mode selection

Abstract

A method and system for performing modal reverberation techniques on an audio signal is described. The method can involve receiving an IR, dividing the IR into a plurality of subbands, determining parameters for each of the modes contained in each subband using a parametric estimation algorithm, aggregating the modes for each of the subbands into a set, and truncating the aggregated set of modes to a subset of modes, thereby simplifying the reverberation effect applied to the audio signal. The reverberation of the audio signal can be manipulated based on the IR, which itself is based on the truncated subset of modes.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application is a continuation of U.S. Patent Application No. 16 / 585,018, filed September 27, 2019, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] Audio engineers, musicians, and even the general public (collectively "users") are familiar with creating and manipulating audio signals. For example, audio engineers edit stereo signals by mixing mono audio signals with each other using effects such as pan and gain to position the mono audio signal within the stereo field. Users also manipulate individual components of audio signals for effect, processing them using multiband structures such as crossover networks for multiband processing. In addition, musicians and audio engineers regularly use audio effects such as compression, distortion, delay, reverberation, and the like to create acoustically pleasing and sometimes unpleasant sounds. Audio signal manipulation is usually performed using specialized software or hardware. The type of hardware and software used to manipulate audio signals generally depends on the user's intent. Users are constantly looking for new ways to create and manipulate audio signals.

[0003] Reverberation is one of the most common effects that users apply to audio signals. The reverberation effect simulates the reverberation of a particular room or acoustic space, so that the audio signal sounds as if it was recorded in a room with a particular impulse response.

[0004] One way to apply reverberation to an audio signal is to use a technique called convolution. Convolution reverberation applies the impulse response of a given acoustic space to the audio signal, so that the audio signal sounds as if it was generated in that given space. However, techniques for manipulating the parameters of convolution reverberation are relatively limited. For example, using convolution reverberation, it may not be possible to isolate and manipulate a single frequency resonance in the audio signal. In addition, using convolution reverberation, it may not be possible to adjust or manipulate a single characteristic of the simulated physical space (e.g., the length of the space, the width of the space).

[0005] An alternative way of applying reverberation to an audio signal is to use a technique called modal reverb. Unlike convolutional reverberation, modal reverberation analyzes the impulse response of a given space, identifies vibration modes in the given space based on this analysis, and then synthesizes the individual vibration modes of the space. As a result, individual frequencies of the reverberation can be edited in isolation, and techniques that manipulate the parameters of modal reverberation are more robust than techniques that manipulate the parameters of convolutional reverberation techniques.

[0006] One drawback of currently known modal reverb techniques is the degree of processing required. A reverberant audio signal often consists of tens of thousands of vibration modes, and modal reverberation techniques must identify and process each of these modes in order to properly restore the reverberation applied to the audio signal. However, typically only about 3000 to 5000 modes can be processed without placing a significant burden on the processor. The amount of processing required can be reduced by discarding modes from the audio signal, but this has the undesirable effect of degrading the quality of the audio signal.

[0007] Another drawback of modal reverberation techniques is that it is difficult to identify all of the modes in an acoustic space. The aforementioned techniques do not provide a high enough resolution to properly identify all of the modes. For example, in some exemplary modal reverberation techniques, the parameters of the modal reverberation can be derived by first transforming the impulse response of the audio signal in the acoustic space into the frequency domain using a Discrete Fourier Transform (DFT), and then identifying the peaks of the transformed signal as the modes of the room. However, the resolution of DFT-based modal identification is low. As a result of the low resolution, the simulated physical space can only be approximated and cannot be easily scaled. Overall, DFT-based modal reverberation techniques can provide some manipulability of the audio signal, but have degraded quality and inaccurate scalability. Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure improves upon known convolutional reverberation techniques by introducing algorithms that provide high-resolution estimates of the modes of an acoustic space through analysis of impulse response (IR) recordings of the space. [Means for solving the problem]

[0009] The algorithm does this by splitting the recording into multiple sub-bands and then estimating the frequency and damping parameters of each mode separately using a parametric estimation algorithm such as ESPRIT. The singular value decomposition (SVD) calculations performed by the ESPRIT algorithm are approximately cube-proportional to the number of modes. This makes the ESPRIT algorithm unwieldy for the large number of modes present in standard acoustic spatial impulse response recordings. However, when the spatial modes are represented by IR split into separate sub-bands, the ESPRIT algorithm can be applied to each sub-band separately, thus reducing the processing normally required for the algorithm. The modal parameters estimated by ESPRIT achieve a higher resolution than traditional DFT-based techniques. This allows the user to, for example, discriminate between frequency-overlapping spatial modes that commonly occur in IR recordings.

[0010] The technique can also be implemented with recordings other than impulse responses: for example, audio recordings of drum sounds can also be analyzed as multiple modes, and such division of such recordings into sub-bands similarly allows the ESPRIT algorithm to be applied in the analysis, as well as modifying the recording based on modal parameters with higher resolution than traditional DFT-based techniques.

[0011] The above technique can be further improved. For example, the sub-bands can be further divided non-uniformly so that the modes are approximately evenly divided between the sub-bands. Firstly, this has the advantage of reducing the processing required for the reasons mentioned above. In addition, the non-uniform division can improve the resolution of the algorithm. For example, a spatial IR may have a relatively high density of modes in one part of the frequency spectrum and a relatively low density of modes in another part of the frequency spectrum. By selecting a relatively narrow sub-band for the part of the audio spectrum with a high density of modes, the resolution of the algorithm applied to the modes within the sub-band can be improved. Similarly, the part of the spectrum with a low density of modes can tolerate a lower resolution and therefore a wider sub-band can be chosen to apply the algorithm.

[0012] One aspect of the present disclosure provides a method for generating a modal reverb effect for manipulating an audio signal. The method may include receiving an impulse response of an acoustic space, the impulse response including a plurality of vibration modes of the acoustic space, dividing the impulse response into a plurality of sub-bands, each sub-band of the impulse response including a portion of the plurality of modes, determining for each sub-band parameters of a portion of the modes included in the sub-band using a parametric estimation algorithm, aggregating the modes of each of the plurality of sub-bands into a set, and truncating the aggregated set of modes into a subset of modes. The method may further include manipulating the audio signal based on the generated modal reverb effect.

[0013] In some examples, instead of receiving an impulse response of an acoustic space, an audio signal can be received. The audio signal can itself contain multiple vibration modes. The remaining steps of the method can then be applied to the audio signal, whereby the audio signal can be split into sub-bands, analyzed using parametric algorithms, etc., so that the modes of the audio signal can be truncated to obtain a result, thereby generating a modified audio signal. Thus, although the present disclosure provides an example of an analysis of an "impulse response", those skilled in the art will recognize and appreciate that the same types of analysis and principles can be applied to other audio signals, and that the examples herein are understood and expected to be applicable to audio signals as well.

[0014] In some examples, the impulse response may be divided into a number of non-uniform sub-bands. Dividing the impulse response into a number of sub-bands may involve passing the impulse response through a filter bank. For each sub-band signal, a number of modes included in the mode portion of the sub-band signal may be estimated. The filter bank may include one or more complex filters, and for each sub-band, each of the sub-bands may have a passband width and a partition width narrower than the passband width. The number of modes may be estimated within the passband width. Determining the parameters of each of the modes included in the sub-band signal may be performed only for the modes within the partition width.

[0015] In some examples, the method can further involve estimating, for each sub-band, a number of modes included in the fraction of modes of that sub-band.

[0016] In some examples, the order of the model of the parametric estimation algorithm applied to the sub-band may be based on the number of the estimated modes contained in the modal fraction of the sub-band.

[0017] In some examples, estimating a number of modes included in the modal fraction of the sub-band may involve determining a peak selection threshold for the sub-band and determining a number of peaks detected in the sub-band that are greater than the peak selection threshold. The estimated number of modes may be based on the determined number of peaks.

[0018] In some examples, the sub-bands may be derived from a Discrete Fourier Transform (DFT) of the impulse response, and determining the peak selection threshold for the sub-bands may involve detecting a maximum peak amplitude for the sub-bands and detecting a minimum peak amplitude for the sub-bands, and the peak selection threshold may be determined based at least in part on the maximum peak amplitude and the minimum peak amplitude.

[0019] In some examples, the peak selection threshold is t = M max - a(M max - M min ) can be calculated based on M max may be the maximum peak amplitude, M min may be the minimum peak amplitude, and a may be a predetermined value between 0 and 1.

[0020] In some examples, determining, for each sub-band, the parameters of each of the portions of the modes may involve determining, for each sub-band to which the parametric estimation algorithm is applied, one or more of a frequency, a decay time, an initial amplitude, or an initial phase of the portion of the mode contained in that sub-band.

[0021] In some examples, determining, for each sub-band, the parameters of each of the portions of the modes may further involve estimating the complex amplitude of each mode included in that sub-band.

[0022] In some examples, the sub-bands may be derived from a Discrete Fourier Transform (DFT), and for each mode contained in the sub-band signal, estimating the complex amplitude may involve minimizing an approximation error of each of the estimated complex amplitudes of the sub-band signal.

[0023] In some examples, the approximation error can be minimized only for modes of the sub-band signals that fall within a passband of a corresponding spectral filter. A different spectral filter can correspond to each of the sub-band signals, and the different spectral filters can cover the audible spectrum without overlap.

[0024] In some examples, the parametric estimation algorithm may be the ESPRIT algorithm.

[0025] In some examples, for each sub-band, determining the parameters of each of the portions of the modes may involve determining a peak selection threshold for that sub-band, and the parameters may be determined for the modes that are included in the portion of the modes and that may have amplitudes greater than the peak selection threshold.

[0026] In some examples, truncating the set into a subset of modes may involve determining, for each of the modes in the set, a signal-to-mask ratio (SMR) for the mode based on a predefined masking curve, and one or more of the modes in the set may be truncated based on the determined SMR.

[0027] In some examples, the step of truncating the set into a subset of modes may further involve receiving an input indicating a total number of modes, the total number of modes being less than or equal to a number of modes included in the set, and truncating the set into a subset of modes having a number of modes equal to the total number of modes.

[0028] In some examples, truncating the set into a subset of modes can further involve sorting the modes included in the set according to the SMR of each mode, where each mode included in the subset can have an SMR greater than the SMR of each mode excluded from the subset.

[0029] In some examples, the predetermined masking curve may be based on a psychoacoustic model.

[0030] Another aspect of the present disclosure provides a system for generating a modal reverberation effect for manipulating an audio signal, the system may include a memory for storing an impulse response and one or more processors. The one or more processors may be configured to: receive an impulse response of an acoustic space, the impulse response including a plurality of modes of vibration of the acoustic space; divide the impulse response into a plurality of sub-bands, each sub-band of the impulse response including a portion of the plurality of modes; estimate, for each sub-band, a number of modes included in the portion of modes of the sub-band; determine parameters of each of the portions of the modes included in the sub-band using a parametric estimation algorithm; aggregate the modes of each of the plurality of sub-bands into a set; and truncate the aggregated set of modes into a subset of modes.

[0031] The above aspects, features and advantages of the present invention will be further understood upon consideration of the following description of exemplary embodiments and the accompanying drawings, in which like reference numerals represent like elements. In describing the embodiments of the present invention illustrated in the drawings, specific terminology may be used for the sake of clarity. However, it is not intended that aspects of the present invention be limited to the specific terminology used. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a block diagram of an example system according to an aspect of the present disclosure. [Diagram 2] FIG. 2 is a flow diagram of an exemplary method according to an aspect of the present disclosure. [Diagram 3] 3 is a flow diagram of an exemplary subroutine of the method shown in FIG. 2. [Figure 4] FIG. 2 illustrates a filterbank according to one aspect of the present disclosure. [Diagram 5] 3 is a flow diagram of another example subroutine of the method shown in FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] FIG. 1 illustrates an example system 100 for performing the modal reverb and mode selection techniques described in this application. The system 100 may include one or more processing devices 110 configured to execute a set of instructions or executable programs. The processor may be a general-purpose CPU or a dedicated component such as an application specific integrated circuit ("ASIC") or other hardware-based processor. Although not required, specialized hardware components may be included to perform certain computational processes faster and more efficiently. For example, the operations of the present disclosure may be performed in parallel on a computer architecture having multiple cores with parallel processing capabilities.

[0034] The various instructions are described in more detail with respect to the flow diagrams of Figures 2, 3 and 5. The system may further include one or more storage devices or memories 120 that store the instructions 130 and programs executed by the one or more processors 110. In addition, the memory 120 may be configured to store data 140, such as one or more IRs 142 and one or more modes 144 identified from the IRs. For example, the IRs 142 may be selected by a user who wishes to apply a reverberation effect to an audio signal. The reverberation effect may be applied by identifying and synthesizing a mode 144 of the selected IR (e.g., multiple modes of the room that generate an IR when the audio signal is played in the room). The data may further include information regarding multiple modes of the space. For simplicity, these modes are also referred to herein as "modes of IRs". As described below, the information regarding the modes may be estimated using an algorithm included in the instructions 130.

[0035] The system 100 may further include an interface 150 for input and output of data. For example, an IR of a given acoustic space may be input to the system via the interface 150, and a selected number of modes or corresponding exponentially damped sinusoids (EDS) and their parameters may be output via the interface 150. Alternatively or in addition, one or more processors may be capable of performing a reverberation operation, where a user may input desired reverberation parameters via the interface 150, and a modified audio signal based on these reverberation parameters may be generated and output via the interface 150. Other parameters and instructions may be provided to and from the system via the interface 150. For example, the number of modes identified in the IR may be a variable entered by the user. This may be used to vary the processing speed of the reverberation operation according to user preferences. The desired number of modes may be preset and stored in the memory 140, or may be entered by the user via the interface 150, or both.

[0036] In some examples, the system 100 may include a user's personal computer, laptop, tablet, or other computing device, within which both the processor and memory are housed. Operations performed by the system are described in more detail with respect to the routines of Figures 2, 3, and 5.

[0037] FIG. 2 is a flow diagram illustrating an example routine 200 .

[0038] In block 210, the system receives an IR of a given space. This space can be a real space (whereby the IR can be a recording in response to an impulse played in the real space) or a simulated or virtual space. The IR can be decomposed into the vibrational modes of each of the IR-simulated spaces, which can be isolated and individually altered. A typical IR can contain more than about 10,000 modes.

[0039] In block 220, the system can divide the IR into multiple sub-bands. For example, the IR modes can be centered on various frequencies across a wide band of frequencies, typically in the audible frequency range (usually considered to be about 20 Hz to about 20 kHz). This band can be broken down into multiple sub-bands, each having a smaller bandwidth than the full band of the IR. In some examples, the sub-bands can be chosen to be non-overlapping, or chosen such that all of the frequencies in the full band of the IR are considered, or both. If both considerations are met, the sum of the bandwidths of the sub-bands can be equal to the bandwidth of the full IR.

[0040] In some examples, the sub-bands can be chosen to have uniform bandwidths in either a logarithmic or non-logarithmic scale. For example, if the IR is decomposed into three sub-bands, each sub-band can have an equal bandwidth. In other examples, the IR can be split into sub-bands based on different factors, which may result in non-uniformity in the bandwidths of the sub-bands. For example, the sub-band splitting can be configured to split the modes of the complete IR approximately evenly.

[0041] In some examples, splitting the full IR may involve first downsampling the full IR using one or more filterbanks, which may be configured to pass some portions of the IR, thereby filtering the IR into different subbands.

[0042] Additionally, in some examples, downsampling can be performed using one or more complex filters that can retain only the positive frequency spectrum of the IR, thereby omitting unwanted portions of the filtered IR from subsequent processing operations.

[0043] At block 230, the number of modes in each sub-band is estimated. The estimated number of modes can inform whether the sub-bands are evenly divided. Additionally or alternatively, the estimated number of modes can inform the desired resolution of subsequent operations of the routine.

[0044] One exemplary subroutine 300 for estimating the number of modes in a given subband is shown in the flow diagram of FIG.

[0045] At block 310, a peak selection threshold for the sub-band may be determined. In some examples, the peak selection threshold may be a fixed value, such as an amplitude value representing the lowest audible loudness. The amplitude values ​​for the sub-band at the sampled frequencies may be determined (e.g., using Fourier transform methods) and then compared to the peak selection threshold, such that only values ​​equal to or greater than the peak selection threshold are determined to be modes of IR.

[0046] In some examples, the peak selection threshold may be determined based on characteristics of the sub-band itself. For example, in block 312, the sub-bands may be derived in the frequency domain using a Discrete Fourier Transform (DFT). Then, in block 314, a maximum peak amplitude of the DFT of the sub-band may be determined, and in block 316, a minimum peak amplitude of the DFT of the sub-band may be determined. In block 318, a peak selection threshold is set based on the maximum and minimum peaks. For example, the formula: t = M max - a(M max - M min ) can be used to set the peak selection threshold t, where M max is the maximum peak amplitude, and M min is the minimum peak amplitude, and a is a predetermined value between 0 and 1. The predetermined value of a may be 0.25.

[0047] In block 320, the number of peaks detected in the sub-bands having amplitudes greater than the peak selection threshold value is counted. The remaining peaks in the DFT are ignored as insignificant or inaudible. The number of peaks counted corresponds to the number of estimated modes in the sub-band. In other words, each counted peak represents the center frequency of a mode identified and counted in the sub-band and used in further processing steps. The remaining modes are not counted and are omitted from further processing steps.

[0048] In block 330, the complete IR can be divided into sub-bands based on the number of peaks detected. This may result in non-uniform sub-bands. To achieve this result, an audio FFT filter bank can be used. Each sub-band is filtered by a causal N-tap finite impulse response (FIR) filter h r [n]:

number

[0049] Where:

number

[0050] Windowing methods known in the art make it possible to design FIR filters by truncating IIR filters. The truncation operation expands the bandwidth of the FIR (compared to an IIR filter), which results in subband filters overlapping in frequency, as shown in FIG. 4. The bandwidth of each FIR filter is constant over its section and starts to roll off as it approaches the edge of its section. This means that modes outside the section are attenuated and their estimation becomes difficult. For any given subband, modes that are within the passband of that subband but outside the section will inevitably be estimated. However, since these modes are inevitably included within the sections of the neighboring passbands, they can be appropriately pruned or ignored, and therefore more robustly estimated.

[0051] In one example of a filter bank designed using the windowing method, we first apply R brickwall filters to all of the frequency responses H r The sum of R filters can be chosen to be 1. The inverse DTFT of R filters is

number

number

[0052] The time-domain multiplication by w[n] performs a convolution between the ideal channel filter in the frequency domain and the window. This results in a frequency-domain spreading of the filter, and the filter responses overlap each other in frequency. This results in a filter bank similar to the one shown in Figure 4.

[0053] FIG. 4 shows the subbands of a filter bank with a passband 410 having a given passband width. The passband width can be used to estimate the number of modes contained in the subband (described in more detail above). The passband can also have a partition 420 with a given partition width. The partition can be used to omit modes from the subband that have center frequencies outside the partition width. It should be appreciated that each partition area spans the original boundary of the corresponding r-th brickwall filter.

[0054] In the example of Figure 4, a particular filter bank is designed using a Chebychev window, however, other windowing techniques known in the art can be used to create other usable filter banks in accordance with this disclosure.

[0055] Returning to FIG. 2, in block 240, a parametric estimation algorithm can be used to determine the parameters of each of the portions of modes contained in the sub-bands. This can be done for each sub-band. One such parametric estimation algorithm that can be applied is the ESPRIT algorithm, which can be used to find the frequency and decay parameters of an exponentially decaying sinusoid (EDS). This algorithm exploits the rotational invariance property of complex sinusoids to determine the complex modal solution of a vector matrix representing the signal vector of a signal.

[0056] Since the above vector matrix is ​​in an m-dimensional space (where m is the number of complex modes), the processing required to solve for the complex modes increases exponentially as the number of modes increases. In other words, the order of the model in the ESPRIT algorithm corresponds to the number of modes estimated to be contained in the sub-bands. This makes processing the entire IR in a single matrix unwieldy. However, the amount of processing can be significantly reduced by splitting the IR into sub-bands and then applying the ESPRIT algorithm to those sub-bands individually, instead of collectively to all of the IR's modes, and solving only for modes with amplitudes greater than a peak selection threshold.

[0057] For a given subset of modes (e.g., modes in a given subband), the complex amplitude of each mode can be estimated by minimizing the following function of a, the matrix of complex amplitudes of the modes:

number

number

number

[0058] Alternatively, the process of estimating the amplitude and phase can be performed again using a divide and conquer approach with a spectral filter. In this method, the minimization function:

number

[0059] The bandwidth b of each mode m in the mode subset m This can be done for each of the sub-bands, which can be done using the following formula:

number

[0060] The above formulas can be applied only to modes that fall within the passband of the subband's spectral filter. For example, for the kth spectral filter associated with the kth subband, the amplitude and phase are in the range

number

[0061] In addition, because the estimation of the amplitude and phase of each mode is performed independently for each sub-band, the processing of each sub-band can be done in parallel, and therefore, for computer architectures with multiple cores with parallel processing capabilities, the modal parameter estimation can be accelerated even further.

[0062] The estimated parameters can be stored in the system's memory for further calculations and subsequent applications.

[0063] Continuing with Figure 2, modes of multiple sub-bands may be aggregated or otherwise recombined into a unified set at block 250. The unified set of modes may be truncated at block 260. The result of the truncation may be a subset of modes.

[0064] For example, for each of the modes in the set, a signal to mask ratio (SMR) of the mode is determined based on a predetermined masking curve, and one or more of the modes in the set are truncated based on the determined SMR.

[0065] An exemplary subroutine 500 for truncating a unified set of modes is shown in the flow diagram of FIG.

[0066] In block 510, a masking curve may be defined. In some examples, the masking curve may be predetermined. The masking curve may be used to compare the relative amplitudes of the modes, but not exclusively to each other, but to the curves. The masking curve may be a psychoacoustic model designed to take into account the psychoacoustics of a person who can hear the audio signal. One exemplary psychoacoustic model is the Psychoacoustic Model 1 from the ISO / IEC MPEG1 standard.

[0067] In some examples, the masking curve may involve a tonal masker and a noise masker. In some cases, including psychoacoustic model 1, a single noise masker may be created by summing the contributions of the non-tonal maskers in each critical band of the signal. Alternatively, this sum may be replaced by an average found to more realistically model the masking curve.

[0068] For each mode in the unified set, a signal-to-mask ratio (SMR) may be determined based on the frequency of each given mode in block 520. The SMR values ​​may be stored in a memory of the system.

[0069] In block 530, the modes can be sorted according to the SMR of each mode. Then, in block 540, an input indicating the total number of modes can be received, and in block 550, the unified set of modes can be truncated to a subset of modes having the modes with the highest SMR. The number of modes included in this subset can be equal to the input total number, which can be less than or equal to the total number of vibration modes included in the IR. The result is a subset of modes that excludes the modes that have the least effect on the IR from a psychoacoustic point of view and includes the modes that have the most effect on the IR. This means that manipulation of modal reverberation parameters based on this subset of modes can be perceived by a listener as no different (or with negligible differences) from manipulation of parameters based on the complete set of identified modes of the complete IR.

[0070] Other methods of truncating modes can be used in place of or in conjunction with subroutine 500 of FIG. 5. For example, modes with relatively low amplitudes (e.g., estimated using least squares methods) can be discarded immediately. As a further example, underdamped modes (whose response envelopes grow by themselves) are unstable and can be discarded. Additionally or alternatively, modes can be organized and grouped into clusters using a K-means algorithm to compress the total number of modes.

[0071] In some cases, the ESPRIT algorithm may estimate IRs for a given acoustic space that fall between 6000 and 12000 modes. The number of modes that a user may wish to truncate, 6000-12000, may vary depending on the processing power per computer, or on the acceptable time constraints or target audio quality per user. Subroutine 500 of FIG. 5 provides scalability and flexibility to control these factors (e.g., the time required to manipulate the IR parameters, the quality and precision of the reverberation effect being manipulated). For example, it may be desirable to limit the total number of modes to 2000-3000, or in other cases 3000-5000. A number between 2000 and 5000 may then be entered in block 440, and the ESPRIT estimated modes may be truncated accordingly for subsequent processing steps.

[0072] Returning to FIG. 2, in block 270, the IR can be simplified to include parameters based on only a subset of the modes. The simplified IR can then be used to manipulate the reverberation effect of the audio signal so that the audio signal sounds as if it were played in an acoustic space having the impulse response of the simplified IR. Due to the techniques described herein, the difference between the original IR of the acoustic space and the simplified IR can be negligible or imperceptible to a listener. As mentioned above, the ability of the listener to perceive the difference can be based on several factors, including the amplitudes of the various vibration modes included in the IR, psychoacoustic models, etc.

[0073] More generally, the present disclosure may enable a user to more effectively and efficiently manipulate the reverberation effects of an audio recording or a portion of an audio recording. For example, a user may want to add sound effects to a portion of an audio recording to make the recording sound as if it were played in a target acoustic space, such as a large hall or a small room. In operation, one or more processors receive or otherwise derive an impulse response of the target acoustic space, transform the impulse response into the frequency domain, divide the frequency plot into sub-bands, and then analyze each of the sub-bands, first individually and then in aggregate, to select the most significant modes of the space (e.g., a subset of the modes described above). The impulse response may then be simplified by discarding the remaining portions, which are the less significant modes of the space. The one or more processors may then use the simplified impulse response of the space to manipulate the audio signal. The result is a modified audio recording.

[0074] In this regard, reverberation is just one example of a characteristic of an audio recording that can be modified using a simplified set of vibrational modes, but mode modification is particularly useful for manipulating reverberation. This is in part because the mapping of modes to perceptually significant parameters (room size, decay time) is relatively straightforward, and because the parameters of the mode filter bank can be stably adjusted at audio rates. Other techniques of manipulation of audio signals or recordings may also be more effective in modifying other characteristics of a given signal.

[0075] The above routine operates under the assumption that the IR can be represented using a sum of exponentially decaying sinusoids (EDS). Thus, the selected mode is effectively an estimate of the EDS parameters of the IR, and control of the selected mode approximates control of the IR's individual EDSs individually. This allows a wide variety of audio effects to be achieved for the IR. These audio effects include, but are not limited to, morphing, spatialization, room size scaling, equalization, etc.

[0076] In addition, the above routines generally describe the processing of the impulse response of a selected acoustic space. However, those skilled in the art will appreciate that similar mode selection concepts and algorithms can be applied to other digital inputs, such as audio signals, even if the audio signal is not the impulse response of the selected space. For example, the audio signal may itself contain an impulse response of the acoustic space in which the audio signal is recorded, which impulse response may contain several vibration modes of the recording space that can be identified and selected using the techniques herein. As a further example, the audio recording may be a drum recording that contains several vibration modes, such that application of the ESPRIT algorithm may allow the vibration modes to be individually modified. In this way, the present application can achieve improved resolution of any mode-modifiable audio recording.

[0077] The above examples are described in the context of using the ESPRIT algorithm. However, other algorithms can be used for parameter approximation. More generally, parametric estimation algorithms other than ESPRIT can be used to decompose a signal into individual components (e.g., modes, damped sinusoids, etc.) and then estimate the parameters of each individual component.

[0078] Although the invention has been described herein with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications can be made to the illustrative embodiments and other arrangements can be devised without departing from the spirit and scope of the invention as defined by the appended claims. The claims as originally filed are as follows: Claim 1: receiving an impulse response of an acoustic space, the impulse response including a plurality of vibration modes of the acoustic space; dividing the impulse response into a plurality of sub-bands, each sub-band of the impulse response including a portion of the plurality of modes; determining, for each sub-band, parameters of each of the portions of the modes contained in that sub-band using a parametric estimation algorithm; aggregating the modes of each of the plurality of sub-bands into a set; truncating the aggregated set of modes to a subset of modes; A method for generating a modal reverberation effect that manipulates an audio signal, comprising: Claim 2: The method of claim 1 , wherein the impulse response is divided into a number of non-uniform sub-bands. Claim 3: The method of claim 1 , wherein splitting the impulse response into a plurality of sub-bands comprises passing the impulse response through a filter bank. Claim 4: estimating, for each subband signal, a number of modes contained in the portion of modes of the subband signal; The filter bank includes one or more complex filters, each of which has a passband width and a partition width narrower than the passband width, The number of modes is estimated within the passband width; The method of claim 3 , wherein determining the parameters of each of the modes contained in the subband signal is performed only for the modes within the partition width. Claim 5: The method of claim 1 , further comprising the step of estimating, for each sub-band, a number of modes contained in the fraction of modes of that sub-band. Claim 6: The method of claim 5 , wherein for each subband, an order of the model of the parametric estimation algorithm applied to that subband is based on the number of the estimated modes contained in the fraction of the modes of that subband. Claim 7: The step of estimating the number of modes included in the portion of modes of the sub-band comprises: determining a peak selection threshold for said sub-band; determining a number of peaks detected within the subband that are greater than the peak selection threshold; Including, The method of claim 5 , wherein the number of estimated modes is based on the number of determined peaks. Claim 8: The sub-bands are derived from a Discrete Fourier Transform (DFT) of the impulse response, and determining the peak selection thresholds for the sub-bands comprises: detecting a maximum peak amplitude of said sub-band; detecting a minimum peak amplitude of said sub-band; Including, The method of claim 7 , wherein the peak selection threshold is determined based at least in part on the maximum peak amplitude and the minimum peak amplitude. Claim 9: The peak selection threshold is t = M max - a(M max - M min ) and M max is the maximum peak amplitude, M min 9. The method of claim 8, wherein a is the minimum peak amplitude and a is a predetermined value between 0 and 1. Claim 10: 2. The method of claim 1, wherein determining, for each subband, the parameters of each of the portions of the modes comprises determining, for each subband to which the parametric estimation algorithm is applied, one or more of a frequency, a decay time, an initial amplitude, or an initial phase of the portions of the modes contained in that subband. Claim 11: The method of claim 10 , wherein for each subband, determining the parameters of each of the portions of the modes further comprises estimating a complex amplitude of each mode contained in the subband. Claim 12: 12. The method of claim 11, wherein the subbands are derived from a Discrete Fourier Transform (DFT), and for each mode contained in the subband signal, estimating the complex amplitude comprises minimizing an approximation error of each of the estimated complex amplitudes of the subband signal. Claim 13: 13. The method of claim 12, wherein the approximation error is minimized only for modes of the subband signals that fall within a passband of a corresponding spectral filter, a different spectral filter corresponding to each of the subband signals, the different spectral filters covering the audible spectrum and being non-overlapping. Claim 14: The method of claim 1 , wherein the parametric estimation algorithm is the ESPRIT algorithm. Claim 15: 2. The method of claim 1, wherein for each subband, determining parameters for each of the portions of the modes comprises determining a peak selection threshold for the subband, the parameters being determined for the modes that are included in the portions of the modes and have amplitudes greater than the peak selection threshold. Claim 16: 2. The method of claim 1, wherein truncating the set into a subset of modes comprises, for each of the modes in the set, determining a signal-to-mask ratio (SMR) of the mode based on a predefined masking curve, and one or more of the modes in the set are truncated based on the determined SMR. Claim 17: The step of truncating the set to a subset of modes comprises: receiving an input indicating a total number of modes that is less than or equal to a number of modes included in the set; truncating said set into a subset of modes having a number of modes equal to said total number of modes; 17. The method of claim 16, further comprising: Claim 18: 18. The method of claim 17, wherein truncating the set into a subset of modes further comprises sorting the modes in the set according to the SMR of each mode, each mode in the subset having an SMR greater than the SMR of each mode excluded from the subset. Claim 19: The method of claim 16 , wherein the predetermined masking curve is based on a psychoacoustic model. Claim 20: A memory for storing an impulse response; one or more processors configured to: receive an impulse response comprising a plurality of vibration modes of an acoustic space; divide the impulse response into a plurality of sub-bands, each sub-band of the impulse response comprising a portion of the plurality of modes; estimate, for each sub-band, a number of modes contained in the portion of modes of the sub-band; determine, for each sub-band, parameters of each of the portion of modes contained in the sub-band signal using a parametric estimation algorithm; aggregate the modes of each of the plurality of sub-bands into a set; and truncate the aggregated set of modes to a subset of modes; 1. A system for generating a modal reverberation effect for manipulating an audio signal, comprising:

Claims

1. receiving an impulse response of an acoustic space, the impulse response including a plurality of vibration modes of the acoustic space; dividing the impulse response into a plurality of sub-bands, each sub-band of the impulse response including a portion of the plurality of modes; determining, for each sub-band, parameters of each of the portions of the modes contained in that sub-band using a parametric estimation algorithm; aggregating the modes of each of the plurality of sub-bands into a set; truncating the aggregated set of modes to a subset of modes; Including, 13. The method of claim 12, wherein the step of truncating the set into a subset of modes comprises determining, for each of the modes in the set, a signal-to-masking ratio (SMR) for the mode based on a predefined masking curve, and one or more of the modes in the set are truncated based on the determined SMR.

2. The method of claim 1 , wherein the impulse response is divided into a number of non-uniform sub-bands.

3. The method of claim 1 , wherein splitting the impulse response into a plurality of sub-bands comprises passing the impulse response through a filter bank.

4. estimating, for each sub-band, a number of modes included in the fraction of modes of that sub-band; The filter bank includes one or more complex filters, each of which has a passband width and a partition width narrower than the passband width, The number of modes is estimated within the passband width; The method of claim 3 , wherein determining the parameters of each of the modes contained in the sub-band is performed only for the modes within the partition width.

5. The method of claim 1 , further comprising the step of estimating, for each sub-band, a number of modes contained in the fraction of modes of that sub-band.

6. The method of claim 5 , wherein for each subband, the order of the model of the parametric estimation algorithm applied to that subband is based on the number of the estimated modes contained in the fraction of the modes of that subband.

7. The step of estimating the number of modes included in the portion of modes of the sub-band comprises: determining a peak selection threshold for said sub-band; determining a number of peaks detected within the subband that are greater than the peak selection threshold; Including, The method of claim 5 , wherein the number of estimated modes is based on the number of determined peaks.

8. The sub-bands are derived from a Discrete Fourier Transform (DFT) of the impulse response, and determining the peak selection thresholds for the sub-bands comprises: detecting a maximum peak amplitude of said sub-band; detecting a minimum peak amplitude of said sub-band; Including, The method of claim 7 , wherein the peak selection threshold is determined based at least in part on the maximum peak amplitude and the minimum peak amplitude.

9. The peak selection threshold is t = M ｍａｘ - a (M ｍａｘ - M ｍｉｎ ) is calculated based on M ｍａｘ is the maximum peak amplitude, M ｍｉｎ 9. The method of claim 8, wherein: a is the minimum peak amplitude, and a is a predetermined value between 0 and 1.

10. 2. The method of claim 1, wherein for each subband, determining the parameters of each of the portions of the modes comprises determining, for each subband to which the parametric estimation algorithm is applied, one or more of a frequency, a decay time, an initial amplitude or an initial phase of the portion of the mode contained in that subband.

11. The method of claim 10 , wherein for each subband, determining the parameters of each of the portions of the modes further comprises estimating a complex amplitude of each mode contained in the subband.

12. 12. The method of claim 11, wherein the sub-bands are derived from a Discrete Fourier Transform (DFT), and for each mode contained in the sub-band, estimating the complex amplitude comprises minimizing an approximation error of each of the estimated complex amplitudes of the sub-band.

13. 13. The method of claim 12, wherein the approximation error is minimized only for modes of the sub-bands that fall within a passband of a corresponding spectral filter, a different spectral filter corresponding to each of the sub-bands, the different spectral filters covering the audible spectrum and being non-overlapping.

14. The method of claim 1 , wherein the parametric estimation algorithm is the ESPRIT algorithm.

15. 2. The method of claim 1, wherein for each subband, determining parameters for each of the portions of the modes comprises determining a peak selection threshold for the subband, the parameters being determined for the modes that are included in the portions of the modes and have amplitudes greater than the peak selection threshold.

16. The step of truncating the set to a subset of modes comprises: receiving an input indicating a total number of modes that is less than or equal to a number of modes included in the set; truncating said set into a subset of modes having a number of modes equal to said total number of modes; The method of claim 1 further comprising:

17. 17. The method of claim 16, wherein truncating the set into a subset of modes further comprises sorting the modes included in the set according to the SMR of each mode, each mode included in the subset having an SMR greater than the SMR of each mode excluded from the subset.

18. The method of claim 1 , wherein the predetermined masking curve is based on a psychoacoustic model.

19. A memory for storing an impulse response; one or more processors configured to: receive an impulse response comprising a plurality of vibration modes of an acoustic space; divide the impulse response into a plurality of sub-bands, each sub-band of the impulse response comprising a portion of the plurality of modes; estimate, for each sub-band, a number of modes included in the portion of modes of the sub-band; determine, for each sub-band, parameters of each of the portion of modes included in the sub-band using a parametric estimation algorithm; aggregate the modes of each of the plurality of sub-bands into a set; determine, for each of the modes included in the set, a signal-to-mask ratio (SMR) of the mode based on a predetermined masking curve; and truncate the aggregated set of modes into a subset of modes, wherein one or more of the modes included in the set are truncated based on the determined SMR; 1. A system for generating a modal reverberation effect for manipulating an audio signal, comprising:

Images