Rendering audio signals using virtualized reverberation

Abstract

A computer-implemented method for processing audio. The method includes obtaining a binaural room impulse response (BRIR) of an acoustic space, receiving an input audio signal, separating the input audio signal into a low frequency component and a high frequency component, and dividing the BRIR of the acoustic space into a first portion appearing before a first time and a second portion appearing after the first time. The method further includes generating a first component of an output audio signal based on the high frequency component of the input audio signal and the first portion of the BRIR, generating a second component of the output audio signal based on the high frequency component of the input audio signal and the second portion of the BRIR, generating a third component of the output audio signal based on the low frequency component of the input audio signal and the BRIR, and outputting the output audio signal.

Description

Technical Field

[0001] Various embodiments relate generally to audio processing and, more particularly, to techniques for rendering audio signals using virtualized reverberation. Background Art

[0002] Reverberation refers to the continued presence of sound in an enclosed or semi-enclosed space after the source has emitted the sound. When sound is produced in an acoustic space (such as a room, concert hall, movie theater, etc.), the sound propagates from one or more sources as a series of sound waves. The propagating sound waves travel through the air and reflect off the boundaries of the acoustic space (such as walls, ceiling, and floor) and / or objects included in the acoustic space. As the reflected sound waves continue to bounce off boundary surfaces and other objects within the acoustic space, these reflections mix together and gradually decay over time as the sound energy is absorbed by surfaces and / or objects in the acoustic space.

[0003] In order to create a more immersive audio experience when using headphones or earplugs, the above-mentioned reverberation effect can be artificially added to the audio signal using digital signal processing techniques. For example, in music production, a sound engineer can render an audio signal with reverberation to simulate the effect of listening to music in a specific acoustic environment (such as a concert hall or auditorium). As another example, in movie or game production, an audio signal can be rendered with reverberation to reproduce the sound of an acoustic environment (such as a cave, a corridor, or a specific outdoor space) depicted on the screen. In other examples such as augmented or virtual reality applications, audio signals rendered with reverberation allow users to perceive an enhanced sense of presence and immersion in the augmented or virtual world.

[0004] Various digital signal processing techniques and algorithms that offer a tradeoff between computational efficiency and accuracy can be used to render reverberation to simulate the effect of an acoustic space during playback of an audio signal. Some virtualized reverberation algorithms render reverberation in an audio signal by convolving the audio signal with a room impulse response (RIR). The RIR is the value of a given acoustic space ( For example, The RIR is the time domain acoustic transfer function between a sound source and a receiver in an acoustic space (e.g., a concert hall, an auditorium, etc.). The RIR of an acoustic space can be measured, for example, using loudspeakers and microphones or simulated using acoustic modeling software. One disadvantage of convolving an audio signal with the RIR is that convolving a long reverberation tail including the RIR with the audio signal is computationally very expensive and time consuming when using high-order processing techniques.

[0005] Other techniques divide the RIR into a first window for direct or early arriving sound and a second window for reflected or late arriving sound ( For example,Reverberation tail). The direct sound window in the RIR is convolved with the audio signal using a high-order computationally intensive processing technique. Then, the reflected sound window in the RIR is separately convolved with the audio signal using a low-order computationally less intensive processing technique. However, these additional techniques typically use RIRs derived from a model of the acoustic space because it is difficult and computationally expensive to obtain the direct sound window and the reflected sound window from the measured RIR of the acoustic space. When compared to the measured RIR of the acoustic space, the RIR derived from the model of the acoustic space may be difficult to obtain and / or generate, and typically cannot capture all of the reverberation effects of the acoustic space.

[0006] As explained above, there is a need for more efficient techniques for rendering audio signals with reverberation. Summary of the invention

[0007] Various embodiments of the present disclosure describe a computer-implemented method for processing audio. The method includes: obtaining a binaural room impulse response (BRIR) of an acoustic space, receiving an input audio signal, separating the input audio signal into a low-frequency component and a high-frequency component, and dividing the BRIR of the acoustic space into a first portion occurring before a first time and a second portion occurring after the first time. The method also includes: generating a first component of an output audio signal based on the high-frequency component of the input audio signal and the first portion of the BRIR, generating a second component of the output audio signal based on the high-frequency component of the input audio signal and the second portion of the BRIR, generating a third component of the output audio signal based on the low-frequency component of the input audio signal and the BRIR, and outputting the output audio signal.

[0008] Other embodiments provide, among other things, one or more non-transitory computer-readable media and systems configured to implement the methods set forth above.

[0009] At least one technical advantage of the disclosed techniques over the prior art is that, using the disclosed techniques, measured RIRs of acoustic spaces can be used to add reverberation to audio signals at a relatively low computational cost. Thus, using the disclosed techniques, relatively modest processing power can be used to render reverberation for a large number of sound sources and acoustic spaces based on measured room impulse responses that sound more natural than room impulse responses derived from acoustic space models. These technical advantages provide one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to be able to understand in detail the manner in which the above-mentioned features of various embodiments are realized, a more particular description of the inventive concept briefly outlined above may be made by reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical embodiments of the inventive concept and are therefore not to be considered as limiting the scope in any way and that there are other equally effective embodiments.

[0011] Figure 1 is a block diagram of a computing device according to one or more aspects of various embodiments;

[0012] Figure 2 is a block diagram of a system for measuring a binaural room impulse response of an acoustic space according to one or more aspects of various embodiments;

[0013] Figure 3 shows mid- and high-frequency components of a measured BRIR of an acoustic space according to one or more aspects of various embodiments;

[0014] Figure 4 illustrates a low frequency component of a measured BRIR of an acoustic space according to one or more aspects of various embodiments;

[0015] Figure 5 A block diagram illustrating a technique for processing an audio signal using a virtualized reverberation algorithm based on measuring binaural room impulse responses according to one or more aspects of various embodiments;

[0016] Figure 6 One or more aspects of various embodiments are shown including Figure 5 A block diagram of the convolution process in the technique for processing audio shown in ;

[0017] Figure 7 A block diagram illustrating a technique for processing an audio signal using a virtualized reverberation algorithm based on measuring binaural room impulse responses according to one or more aspects of various embodiments; and

[0018] Figure 8 is a flow chart of method steps for processing an audio signal according to one or more aspects of various embodiments. DETAILED DESCRIPTION

[0019] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0020] Figure 11 is a block diagram of a computing device 100 according to various embodiments. As will be described in more detail below, the computing device 100 can be used to render audio signals using virtualized reverberation techniques. The computing device 100 can be implemented as any type of computing device suitable for practicing various embodiments, such as a desktop computer, a laptop computer, a mobile device, a server, or an intelligent audio output device. As shown, the computing device 100 includes, but is not limited to, one or more processing units 102, a network interface 104, an input / output (I / O) device interface 106, an input device 108, an output device 110, a system storage device 112, and a system memory 114. The computing device 100 also includes an interconnect 116, which is configured to facilitate the transmission of data (such as programming instructions and application data) between the processing unit 102, the network interface 104, the I / O device interface 106, the system storage device 112, and the system memory 114.

[0021] The processing unit 102 may be any technically feasible processing device configured to process data and execute program instructions. For example, the processing unit 102 may include one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), microprocessors, microcontrollers, other types of processing units, and / or combinations of different processing units. The processing unit 102 is configured to retrieve and execute programmed instructions stored in the system memory 114, such as the binaural room impulse response (BRIR) measurement application 118 and the virtualized reverberation application 120. Similarly, the processing unit 102 is configured to store application data ( For example, software library) and retrieve application data from system storage 112 and / or system memory 114.

[0022] Computing device 100 may connect to other devices and / or networks via network interface 104. In some embodiments, network interface 104 is hardware, software, or a combination of hardware and software that is configured to connect to and interface with one or more devices and / or networks. In some embodiments, network interface 104 facilitates communication via one or more standard and / or proprietary protocols ( For example, Bluetooth, proprietary protocols associated with a specific manufacturer, etc.) to communicate with other devices or systems.

[0023] The I / O device interface 106 is configured to receive input data from the input device 108 and transmit the input data to the processing unit 102 via the interconnect 116. For example, the input device 108 may include one or more buttons, a keyboard, a mouse, a graphical user interface, a touch screen display, a microphone, and / or other input devices. The I / O device interface 106 is also configured to receive output data from the processing unit 102 via the interconnect 116 and transmit the output data to the output device 110. For example, the output device 110 may include one or more of a display device, a touch screen display, a graphical user interface, a speaker, and / or other output devices.

[0024] System storage 112 may include non-volatile storage for applications, software modules, and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other magnetic storage devices, optical storage devices, solid-state storage devices, etc. System storage 112 may be located in whole or in part in a remote storage system referred to herein as the "cloud" and accessed through a network connection via network interface 104. System storage 112 is configured to store files such as ( For example, In one example, the system storage device 112 stores binaural room impulse response (BRIR) data 122 associated with measured BRIRs of one or more acoustic spaces. The binaural room impulse response refers to the acoustic impulse response between the sound source and the listener's ears in the acoustic space. Therefore, each BRIR includes a left ear component of the BRIR and a right ear component of the BRIR.

[0025] The system memory 114 may include random access memory (RAM) modules, flash memory units, or any other type of memory unit or combination thereof. One or more processing units 102, network interface 104, and I / O device interface 106 are configured to read data from and write data to the system memory 114. The system memory 114 includes various software programs and modules ( For example, operating system, one or more application programs) and application data associated with the software programs ( For example, Data loaded from system storage 112). For example, as will be described in more detail below, system memory 114 includes a BRIR measurement application 118 and a virtualized reverberation application 120.

[0026] The BRIR measurement application 118 may be executed by the processing unit 102 to measure the BRIR of the acoustic space and store the measured BRIR of the acoustic space as BRIR data 122 in the system storage 112 . Figure 2 is a block diagram of an example system 200 for measuring binaural room impulse responses of an acoustic space 202. The acoustic space 202 may be any type of room or listening environment in which it is desired to simulate a listening experience with a reverberant effect. For example, the acoustic space 202 may be a concert hall, an auditorium, a theater, a stadium, or some other type of listening environment. As shown, the system 200 includes a computing device 100 executing a BRIR measurement application 118, a speaker 204, and an omnidirectional microphone 206. Although the speaker 204 is shown as a single speaker, in some embodiments, the system 200 includes multiple speakers distributed around the acoustic space 202. In some embodiments, the speaker 204 includes multiple speaker drivers. For example, the speaker 204 includes one or more tweeters, a midrange driver, and a woofer.

[0027] In operation, the BRIR measurement application 118 controls the speaker 204 to emit an audio signal within the acoustic space 202. The omnidirectional microphone 206 records the emitted audio signal propagating through the acoustic space 202, and the BRIR measurement application 118 processes the recorded audio signal to determine the BRIR of the acoustic space 202. The BRIR measurement application 118 may measure and determine the BRIR of the acoustic space 202 using one or more known techniques, such as a maximum length sequence (MLS) technique, an inverted repeat sequence (IRS) technique, a time-stretched pulse technique, or a logarithmic sine sweep technique. The BRIR measurement application 118 then stores the measured BRIR of the acoustic space 202 as BRIR data 122 in the system storage device 112. As described above, the BRIR of the acoustic space 202 is an acoustic impulse response between a sound source in the acoustic space 202 and a listener's ear. Therefore, the BRIR of the acoustic space 202 includes a left-ear component of the BRIR and a right-ear component of the BRIR.

[0028] Return to reference Figure 1, when executed by the processing unit 102, the virtualized reverberation application 120 renders the audio signal with reverberation based on the measured BRIR of the acoustic space. For example, the virtualized reverberation application 120 renders the audio signal with reverberation based on the BRIR of the acoustic space 202 measured by the BRIR measurement application 118 and stored as BRIR data 122 in the system storage device 112. Rendering the audio signal with reverberation based on the measured BRIR of the acoustic space includes adding reverberation to the audio signal to simulate the effect of listening to the audio playback in the acoustic space. In some embodiments, when rendering the audio signal with reverberation based on the measured BRIR, the virtualized reverberation application 120 filters or separates the measured BRIR into mid-high frequency components of the measured BRIR and low frequency components of the measured BRIR. As used herein, the term mid-high frequency may be used interchangeably with the term high frequency.

[0029] Figure 3 The mid-high frequency components 300 of the measured BRIR of an acoustic space according to one or more aspects of various embodiments are shown. For example, Figure 3 The mid-high frequency component 300 of the BRIR measured by the BRIR measurement application 118 of the acoustic space 202 is shown. The mid-high frequency component 300 of the measured BRIR includes a component of the measured BRIR having a frequency greater than the crossover frequency. In some embodiments, the value of the crossover frequency is determined empirically. In some embodiments, the value of the crossover frequency is determined based on one or more characteristics of the acoustic space 202. In a non-limiting example, the crossover frequency is 900 Hz. In other examples, the value of the crossover frequency is less than 900 Hz or greater than 900 Hz. Hereinafter, the mid-high frequency component 300 of the measured BRIR of the acoustic space 202 may also be referred to as the high frequency component of the measured BRIR of the acoustic space 202.

[0030] like Figure 3 As shown in FIG. 1 , the mid-high frequency component of the measured BRIR includes a direct sound portion 302 and a reflected sound portion 304. When the BRIR measurement application 118 measures the BRIR of the acoustic space 202, the direct sound portion 302 includes the portion of the sound emitted by the speaker 204 that is directly received by the omnidirectional microphone 206. That is, the direct sound portion 302 is the portion of the sound emitted by the speaker 204 that directly reaches the omnidirectional microphone 206 without being reflected off one or more surfaces ( For example, The time at which the direct sound portion 302 arrives at the omnidirectional microphone 206 indicates the time at which the sound emitted by a source (such as the speaker 204) in the acoustic space 202 directly reaches the listener's ear.

[0031] In contrast, when the BRIR measurement application 118 measures the BRIR of the acoustic space 202, the reflected sound portion 304 includes the portion of the sound emitted by the speaker 204 that is reflected off one or more surfaces of the acoustic space 202 before reaching the omnidirectional microphone 206. The time at which the reflected sound portion 304 arrives at the omnidirectional microphone 206 indicates the time at which the sound emitted by a source (such as the speaker 204) in the acoustic space 202 that reflects off one or more surfaces reaches the eardrum of the listener. The reflected sound portion 304 can be referred to as the "reverberation tail" of the measured BRIR of the acoustic space 202. Figure 3 As shown in , the peak amplitude of the direct sound portion 302 is generally greater than the peak amplitude of the reflected sound portion 304. This occurs because when a sound wave is reflected off a surface of the acoustic space 202, some of the energy of the sound wave is absorbed by the surface of the acoustic space 202, and because the sound wave is attenuated as the sound wave travels a greater distance.

[0032] like Figure 3 As further shown in FIG. 1 , the direct sound portion 302 reaches the omnidirectional microphone 206 before the reflected sound portion 304 reaches the omnidirectional microphone 206. This occurs because the path of the sound waves that reflect off one or more surfaces of the acoustic space 202 before reaching the omnidirectional microphone 206 is shorter than that from the source ( For example, The path of the sound that travels directly from the speaker 204 to the omnidirectional microphone 206 is long. Therefore, when rendering the audio signal with reverberation, the virtualized reverberation application 120 can temporally divide the mid-high frequency component 300 of the measured BRIR of the acoustic space 202 into the direct sound portion 302 and the reflected sound portion 304. That is, the virtualized reverberation application 120 can divide the mid-high frequency component 300 of the measured BRIR into the first time window 306 and the second time window 308 to separate the direct sound portion 302 from the reflected sound portion 304. When compared to other techniques for processing the BRIR of the acoustic space, temporally dividing the measured BRIR of the acoustic space into the first time window and the second time window is computationally cheaper.

[0033] In the example shown, the first time window 306 includes the time before time T1 and the second time window 308 includes the time after time T1. The value of time T1 is selected as the time when the direct sound portion 302 has reached the omnidirectional microphone 206, and is also selected as the time when the reflected sound portion 304 has not yet reached the omnidirectional microphone 206. The value of time T1 depends on one or more characteristics of the acoustic space 202, because the amount of time it takes for the direct sound portion 302 and / or the reflected sound portion 304 to reach the omnidirectional microphone 206 can vary between different acoustic spaces. In some embodiments, the virtualized reverberation application 120 automatically determines the value of time T1 by analyzing the mid-high frequency component 300 of the measured BRIR. In some embodiments, the value of time T1 is determined empirically and / or manually defined by a user.

[0034] In one non-limiting example, the value of time T1 is 2 ms. In this example, the direct sound portion 302 included in the mid-high frequency component 300 for measuring BRIR arrives at the omnidirectional microphone 206 within 2 ms of being emitted by the speaker 204. Continuing with this example, the reflected sound portion 304 included in the mid-high frequency component 300 for measuring BRIR arrives at the omnidirectional microphone 206 more than 2 ms after being emitted by the speaker 204. In some examples, the value of time T1 is less than 2 ms. In other examples, the value of time T1 is greater than 2 ms.

[0035] exist Figure 3 In the example shown, the virtualized reverb application 120 implements a hard windowing technique to divide the mid-high frequency component 300 of the measured BRIR into a first time window 306 occurring before time T1 and a second time window 308 occurring after time T1, thereby separating the direct sound portion 302 from the reflected sound portion 304. When the virtualized reverb application 120 implements a hard windowing technique (where there is no overlap between the first time window 306 and the second time window 308) to separate the direct sound portion 302 from the reflected sound portion 304, there is no negative impact that may generally be attributed to abruptly cutting off the impulse response. In some embodiments, the virtualized reverb application 120 implements a soft windowing technique (where there is a certain overlap between the first time window 306 and the second time window 308) to separate the direct sound portion 302 from the reflected sound portion 304.

[0036] As will be described in more detail below, dividing the mid-high frequency component 300 of the measured BRIR into the first time window 306 and the second time window 308 allows the virtualized reverberation application 120 to render the mid-high frequency component of the audio signal having the direct sound portion 302 using an accurate but computationally expensive processing technique, and to render the mid-high frequency component of the audio signal having the reflected sound portion 304 separately using a less accurate but computationally efficient processing technique. For example, the virtualized reverberation application 120 implements a reverberant loudspeaker (RVL) method to discretely render the mid-high frequency component of the audio signal, wherein the direct sound portion 302 is included in the mid-high frequency component 300 of the measured BRIR. As another example, the virtualized reverberation application 120 implements a low-order processing technique in which the mid-high frequency component of the audio signal is encoded and decoded before rendering the mid-high frequency component of the audio signal, wherein the reflected sound portion 304 is included in the mid-high frequency component of the measured BRIR.

[0037] Figure 4 The low frequency component of the measured BRIR of an acoustic space according to one or more aspects of various embodiments is shown. For example, Figure 4 A low frequency component 400 of the BRIR of the acoustic space 202 measured by the BRIR measurement application 118 is shown. Measuring the low frequency component 400 of the BRIR includes measuring a component of the BRIR having a frequency less than a crossover frequency. In some embodiments, the value of the crossover frequency is determined empirically. In some embodiments, the value of the crossover frequency is determined based on one or more characteristics of the acoustic space 202. In one non-limiting example, the crossover frequency is 900 Hz. In other examples, the value of the crossover frequency is less than 900 Hz or greater than 900 Hz.

[0038] Similar to the mid-high frequency component 300 for measuring the BRIR, the low frequency component 400 for measuring the BRIR also includes a direct sound portion 402 and a reflected sound portion 404. However, unlike the mid-high frequency component 300 for measuring the BRIR, the direct sound portion 402 and the reflected sound portion 404 included in the low frequency component 400 for measuring the BRIR cannot be easily separated in time. Figure 4As shown in FIG. 1 , there is an overlap between the time when the direct sound portion 402 arrives at the omnidirectional microphone 206 and the time when the reflected sound portion 404 arrives at the omnidirectional microphone 206. This overlap in arrival times can be attributed to the longer wavelength of the low frequency sound waves when compared to the wavelength of the mid and high frequency sound waves. Therefore, even if the virtualized reverberation application 120 divides the low frequency component 400 measuring the BRIR into a first time window 406 occurring before time T1 and a second time window 408 occurring after time T1, such windowing does not separate the direct sound portion 402 from the reflected sound portion 404. In some cases, when rendering the low frequency component of the audio signal, attempting to separate the direct sound portion 402 from the reflected sound portion 404 may introduce an undesirable windowing effect, which is not a problem at higher frequencies. Therefore, when rendering the low frequency component of the audio signal, the virtualized reverberation application 120 does not separate the direct sound portion 402 from the reflected sound portion 404.

[0039] Figure 5 A block diagram 500 of a technique for processing an audio signal using a virtualized reverberation algorithm based on measuring binaural room impulse responses according to one or more aspects of various embodiments is shown. The technique can be implemented, for example, by a virtualized reverberation application 120 executing on a processing unit 102 of a computing device 100. In some embodiments, the virtualized reverberation application 120 is implemented using the BRIR of the acoustic space 202 measured by the BRIR measurement application 118 and stored as BRIR data 122 in the system storage device 112. Figure 5 The processing technique shown in .

[0040] As shown, the virtualized reverb application 120 receives an input audio signal to render with reverb. Figure 5 In the example shown, the input audio signal is a monaural audio signal comprising a single audio channel. However, in some embodiments, the input audio signal comprises more than one audio channel. For example, the input audio signal may comprise two or more audio channels.

[0041] The virtualized reverberation application 120 applies a filtering process 502 to the input audio signal to separate the low frequency component of the input audio signal from the mid-high frequency component of the input audio signal. For example, the filtering process 502 includes applying a low pass filter to the input audio signal to obtain the low frequency component of the input audio signal. The low frequency component of the input audio signal includes a component whose frequency is less than the crossover frequency in the input audio signal. The filtering process 502 also includes applying a high pass filter to the input audio signal to obtain the mid-high frequency component of the input audio signal. The mid-high frequency component of the input audio signal includes a component whose frequency is greater than the crossover frequency in the input audio signal. As described above, the value of the crossover frequency can be determined empirically and / or based on one or more characteristics of the acoustic space. In a non-limiting example, the crossover frequency is 900 Hz. Hereinafter, the mid-high frequency component of an audio signal (such as an input audio signal) may also be simply referred to as the high frequency component of the audio signal.

[0042] After separating the low frequency component from the mid-high frequency component of the input audio signal, the virtualized reverb application 120 uses the first processing technique 504 to add reverb to the low frequency component of the input audio signal. In addition, the virtualized reverb application 120 uses the second processing technique 506 and the third processing technique 508 to add reverb to the mid-high frequency component of the input audio signal. When compared to the third processing technique 508, which may be implemented as a discrete RVL method for adding reverb to discrete mid-high frequency components of the input audio signal, the first processing technique 504 and the second processing technique 506 are less precise but more computationally efficient. That is, when implemented by the virtualized reverb application 120, the first processing technique 504 and the second processing technique 506 consume fewer computational resources than the third processing technique 508.

[0043] As shown, the first processing technique 504 includes an encoding and decoding process 510 in which the virtualized reverb application 120 encodes the low-frequency components of a single audio channel of the input audio signal into a plurality of encoded channels of low-frequency audio. When implementing the encoding and decoding process 510, the virtualized reverb application 120 further decodes the plurality of encoded channels of low-frequency audio into a plurality of decoded channels of low-frequency audio. In some cases, the plurality of decoded channels of low-frequency audio corresponds to the cardinal directions ( For example, Hereinafter, the decoded channels of low frequency audio output by the encoding and decoding process 510 may be referred to as converted channels of low frequency audio and / or converted low frequency audio channels.

[0044] In the example shown, the virtualized reverb application 120 uses a low-order ( For example,First-order) Ambisonics encoder-decoder to encode a single audio channel of an input audio signal into four low-frequency Ambisonics coded audio channels ( For example, The Ambisonics B format) is used to decode the four low-frequency Ambisonics encoded audio channels into six low-frequency audio channels. Each of the six low-frequency decoded or converted audio channels corresponds to a respective direction of sound arriving at the listener's ears ( For example, Those skilled in the art will appreciate that in some embodiments, the virtualized reverb application 120 uses other types of low-order encoder-decoders to implement the encoding and decoding process 510. Additionally, those skilled in the art will appreciate that in some embodiments, the encoding and decoding process 510 encodes and decodes a single audio channel of an input audio signal ( For example, Convert) to less than or more than six channels of low-frequency audio.

[0045] The first processing technique 504 also includes a gain reduction process 512, in which the virtualized reverb application 120 reduces the gain of the converted channel of the low frequency audio. Figure 5 In the example shown, the gain reduction process 512 reduces the gain of six low-frequency conversion audio channels. The virtualized reverberation application 120 applies the gain reduction process 512 to multiple conversion low-frequency audio channels to prevent rendering of an audio signal with reverberation, wherein the mid-high frequency components of the rendered audio signal are suppressed by the low-frequency components. In a non-limiting example, the gain reduction process 512 reduces the corresponding gains of multiple conversion low-frequency audio channels to a value between -8dB and -15dB. However, those skilled in the art will appreciate that in some embodiments, the gain reduction process 512 reduces the corresponding gains of multiple conversion low-frequency audio channels to other values.

[0046] The virtualized reverberation application 120 then applies a low-frequency convolution process 514 to the plurality of converted low-frequency audio channels to generate a left low-frequency audio channel and a right low-frequency audio channel including a reverberation effect. The low-frequency convolution process 514 includes convolving the plurality of converted low-frequency audio channels with the low-frequency components of the measured BRIR of the acoustic space. As described above, because it is difficult to perform such a process in a computationally efficient manner ( For example, The low-frequency component of the measured BRIR is divided into a direct sound part and a reflected sound part (in time), so the low-frequency convolution process 514 convolves the plurality of converted low-frequency audio channels with both the direct sound part and the reflected sound part included in the low-frequency component of the measured BRIR. For example, when implementing the low-frequency convolution process 514, the virtualized reverberation application 120 convolves the plurality of converted low-frequency audio channels with the direct sound part 402 and the reflected sound part 404 included in the low-frequency component 400 of the measured BRIR of the acoustic space 202.

[0047] Figure 6 A block diagram 600 of a convolution process that can be used to implement the low frequency convolution process 514 is shown. Figure 6 During the low-frequency convolution process 514, the virtualized reverb application 120 convolves each of the plurality of converted low-frequency audio channels with a corresponding low-frequency left-ear component of the measured BRIR. When the corresponding converted low-frequency audio channel is convolved with the corresponding low-frequency left-ear component of the measured BRIR, the result is a rendered low-frequency audio channel that includes a reverberation effect. After convolving each of the converted low-frequency audio channels with the corresponding low-frequency left-ear component of the measured BRIR, the virtualized reverb application 120 adds the resulting rendered low-frequency audio channels into a single rendered low-frequency left-ear audio channel that includes a reverberation effect. In some embodiments, each of the low-frequency left-ear components of the measured BRIR corresponds to a corresponding direction ( For example, up, down, left, right, front, back).

[0048] Similarly, during the low-frequency convolution process 514, the virtualized reverb application 120 convolves each of the plurality of converted low-frequency audio channels with a corresponding low-frequency right-ear component of the measured BRIR. When the corresponding converted low-frequency audio channel is convolved with the corresponding low-frequency right-ear component of the measured BRIR, the result is a rendered low-frequency audio channel that includes a reverberation effect. After convolving each of the converted low-frequency audio channels with the corresponding low-frequency right-ear component of the measured BRIR, the virtualized reverb application 120 sums the resulting rendered low-frequency audio channels into a single rendered low-frequency right-ear audio channel that includes a reverberation effect. In some embodiments, each of the low-frequency right-ear components of the measured BRIR corresponds to a corresponding direction ( For example, up, down, left, right, front, back).

[0049] As described above, after the virtualized reverb application 120 applies the low frequency convolution process 514 to the plurality of converted low frequency audio channels, the result is a rendered low frequency left ear audio channel and a rendered low frequency right ear audio channel. The first processing technique 504 also includes a headphone equalization process 516, in which the virtualized reverb application 120 equalizes the rendered low frequency left ear audio channel and the rendered low frequency right ear audio channel. The headphone equalization process 516 implements one or more known equalization techniques to adjust the respective volume levels of the frequency components included in the rendered low frequency left ear audio channel and the rendered low frequency right ear audio channel.

[0050] Return to reference Figure 5 The virtualized reverb application 120 also uses a second processing technique 506 and a third processing technique 508 to add reverb to the mid- and high-frequency components of the input audio signal. Figure 5 As shown in , the second processing technique 506 includes an encoding and decoding process 518, in which the virtualized reverb application 120 encodes the mid-high frequency components of a single audio channel of the input audio signal into a plurality of encoded channels of mid-high frequency audio. When implementing the encoding and decoding process 518, the virtualized reverb application 120 further decodes the plurality of encoded channels of mid-high frequency audio into a plurality of decoded channels of mid-high frequency audio. In some cases, the plurality of decoded channels of mid-high frequency audio corresponds to the cardinal directions ( For example, Hereinafter, the decoded channels of mid-high frequency audio output by the encoding and decoding process 510 may be referred to as converted channels of mid-high frequency audio and / or converted mid-high frequency audio channels.

[0051] In the example shown, the virtualized reverb application 120 uses a low-order ( For example, First-order) Ambisonics encoder-decoder to encode a single audio channel of an input audio signal into four mid- and high-frequency Ambisonics coded audio channels ( For example, The Ambisonics B format) is used to decode the four mid-high frequency Ambisonics encoded audio channels into six mid-high frequency audio channels. Each of the six mid-high frequency decoded or converted audio channels corresponds to a respective direction of sound arriving at the listener's ear ( For example, Up, down, left, right, front, back). However, those skilled in the art will appreciate that in some embodiments, the virtualized reverb application 120 uses other types of low-order encoder-decoders to implement the encoding and decoding process 518. Additionally, those skilled in the art will appreciate that in some embodiments, the encoding and decoding process 518 encodes and decodes a single audio channel of an input audio signal ( For example, Convert) into less than or more than six channels of mid- and high-frequency audio.

[0052] The second processing technique 508 also includes a gain reduction process 520 in which the virtualized reverb application 120 reduces the gain of the multiple converted channels of the mid- and high-frequency audio. Figure 5In the example shown, the gain reduction process 520 reduces the gain of six mid-high frequency conversion audio channels. The virtualized reverberation application 120 applies the gain reduction process 520 to multiple conversion low-frequency audio channels to prevent rendering of an audio signal with reverberation, wherein the mid-high frequency components rendered using the direct sound portion of the measured BRIR are suppressed by the mid-high frequency components rendered using the reflected sound portion of the measured BRIR. In a non-limiting example, the gain reduction process 520 reduces the corresponding gains of multiple conversion mid-high frequency audio channels to values ​​between -8dB and -15B. However, those skilled in the art will appreciate that in some embodiments, the gain reduction process 520 reduces the corresponding gains of multiple conversion mid-high frequency audio channels to other values.

[0053] Then, the virtualized reverb application 120 applies a reflection convolution process 522 to the plurality of converted mid-high frequency audio channels to generate a left audio channel and a right audio channel of a reverberation effect including reflected sound that reaches the listener's ears. The reflection convolution process 522 includes convolving the plurality of converted mid-high frequency audio channels with only the reflected sound portion included in the mid-high frequency component of the measured BRIR of the acoustic space. For example, when implementing the reflection convolution process 522, the virtualized reverb application 120 convolves the plurality of converted mid-high frequency audio channels with only the reflected sound portion 304 included in the mid-high frequency component 300 of the measured BRIR of the acoustic space 202. As described above with respect to Figure 3 As described, the reflected sound portion 304 included in the mid-high frequency component 300 for measuring the BRIR may be separated in time from the direct sound portion 302 included in the mid-high frequency component 300 .

[0054] Figure 6 The block diagram 600 of the convolution process shown in FIG. 5 can also be used to implement the reflection convolution process 522. Figure 6 During the reflection convolution process 522, the virtualized reverb application 120 convolves each of the plurality of converted mid-high frequency audio channels with a corresponding reflected sound portion included in the mid-high frequency left ear component of the measured BRIR. When the corresponding converted mid-high frequency audio channel is convolved with the corresponding reflected sound portion included in the mid-high frequency left ear component of the measured BRIR, the result is a rendered mid-high frequency audio channel that includes a reverberation effect associated with the reflected sound portion of the measured BRIR. After convolving each of the converted mid-high frequency audio channels with the corresponding reflected sound portion included in the mid-high frequency left ear component of the measured BRIR, the virtualized reverb application 120 sums the resulting rendered mid-high frequency audio channels into a single rendered mid-high frequency left ear audio channel that includes a reverberation effect associated with the reflected sound portion of the measured BRIR. In some embodiments, each of the reflected sound portions included in the mid-high frequency left ear component of the measured BRIR corresponds to a corresponding direction ( For example, up, down, left, right, front, back).

[0055] Similarly, during the reflection convolution process 522, the virtualized reverb application 120 convolves each of the plurality of converted mid-high frequency audio channels with a corresponding reflected sound portion included in the mid-high frequency right ear component of the measured BRIR. When the corresponding converted mid-high frequency audio channel is convolved with the corresponding reflected sound portion included in the mid-high frequency right ear component of the measured BRIR, the result is a rendered mid-high frequency audio channel that includes a reverberation effect associated with the reflected sound portion of the measured BRIR. After convolving each of the converted mid-high frequency audio channels with the corresponding reflected sound portion included in the mid-high frequency right ear component of the measured BRIR, the virtualized reverb application 120 sums the resulting rendered mid-high frequency audio channels into a single rendered mid-high frequency right ear audio channel that includes a reverberation effect associated with the reflected sound portion of the measured BRIR. In some embodiments, each of the reflected sound portions included in the mid-high frequency right ear component of the measured BRIR corresponds to a corresponding direction ( For example, up, down, left, right, front, back).

[0056] As described above, after the virtualized reverb application 120 applies the reflection convolution process 522 to the plurality of converted mid-high frequency audio channels, the result is a rendered mid-high frequency left ear audio channel including a reverb effect associated with the reflected sound portion of the measured BRIR and a rendered mid-high frequency right ear audio channel including a reverb effect associated with the reflected sound portion of the measured BRIR. The second processing technique 506 also includes a headphone equalization process 524, in which the virtualized reverb application 120 equalizes the above-mentioned rendered mid-high frequency left ear audio channel and the rendered mid-high frequency right ear audio channel including the reverb effect associated with the reflected sound portion of the measured BRIR. The headphone equalization process 524 (which is similar to the headphone equalization process 516) implements one or more known equalization techniques to adjust the respective volume levels of the frequency components included in the rendered mid-high frequency left ear audio channel and the rendered mid-high frequency right ear audio channel.

[0057] Regarding the third processing technique 508, the virtualized reverb application 120 does not encode and decode or convert the mid-high frequency components of a single audio channel included in the input audio signal before implementing the direct convolution process 526. Instead, the virtualized reverb application 120 renders the discrete mid-high frequency components of a single audio channel included in the input audio signal with reverberation using the high-resolution direct convolution process 526. Figure 5 In the example shown, the third processing technique 508 is implemented using the RVL method for adding reverberation to the audio signal.

[0058] When applying the direct convolution process 526 to the discrete mid-high frequency components of a single audio channel included in the input audio signal, the virtualized reverberation application 120 convolves the discrete mid-high frequency audio channel with only the direct sound portion included in the mid-high frequency components of the measured BRIR of the acoustic space. For example, when implementing the direct convolution process 526, the virtualized reverberation application 120 convolves the discrete mid-high frequency audio channel with only the direct sound portion 302 included in the mid-high frequency components 300 of the measured BRIR of the acoustic space 202. As described above with respect to Figure 3 As described, the direct sound portion 302 included in the mid-high frequency component 300 for measuring the BRIR may be separated in time from the reflected sound portion 304 included in the mid-high frequency component 300 .

[0059] In one example, when implementing the direct convolution process 526, the virtualized reverb application 120 convolves the discrete mid-high frequency audio channels with the direct sound portion included in the mid-high frequency left ear component of the measured BRIR. The direct sound portion included in the mid-high frequency left ear component of the measured BRIR corresponds to the direction in which the sound emitted by the source directly reaches the left ear of the listener. Similarly, when implementing the direct convolution process 526, the virtualized reverb application 120 also convolves the discrete mid-high frequency audio channels with the direct sound portion included in the mid-high frequency right ear component of the measured BRIR. The direct sound portion included in the mid-high frequency right ear component of the measured BRIR corresponds to the direction in which the sound emitted by the source directly reaches the right ear of the listener.

[0060] The result of the direct convolution process 526 is a rendered mid-high frequency left ear audio channel including a reverberation effect associated with the direct sound portion of the measured BRIR and a rendered mid-high frequency right ear audio channel including a reverberation effect associated with the direct sound portion of the measured BRIR. Using a high resolution direct convolution process 525 to convolve the discrete mid-high frequency audio channels with the direct sound portion included in the mid-high frequency component of the measured BRIR helps to offset potential performance losses due to the use of low-order encoding and decoding in the second processing technique 506. As described above, in some examples, a low-order Ambisonics encoder-decoder is used to implement the low-order encoding and decoding in the second processing technique 506.

[0061] The third processing technique 508 also includes a headphone equalization process 528, in which the virtualized reverb application 120 equalizes the rendered mid-high frequency left ear audio channel and the rendered mid-high frequency right ear audio channel including the reverb effect associated with the direct sound portion of the measured BRIR. The headphone equalization process 528 (which is similar to the headphone equalization processes 516 and 524) implements one or more known equalization techniques to adjust the respective volume levels of the frequency components included in the rendered mid-high frequency left ear audio channel and the rendered mid-high frequency right ear audio channel.

[0062] Then, the virtualized reverb application 120 implements a summing process 530 to combine the rendered left-ear audio channels generated using the first processing technique 504, the second processing technique 506, and the third processing technique 508, and to combine the rendered right-ear audio channels generated using the first processing technique 504, the second processing technique 506, and the third processing technique 508 into a rendered stereo audio signal including a left audio channel and a right audio channel. For example, when implementing the summing process 530, the virtualized reverb application 120 sums the rendered low-frequency left-ear audio channel generated using the first processing technique 504, the rendered mid-high frequency left-ear audio channel generated using the second processing technique 506, and the rendered mid-high frequency left-ear audio channel generated using the third processing technique 508 to obtain a single left-ear audio channel including a reverb effect. Similarly, when implementing the summing process 530, the virtualized reverb application 120 sums the rendered low-frequency right ear audio channel generated using the first processing technique 504, the rendered mid-high frequency right ear audio channel generated using the second processing technique 506, and the rendered mid-high frequency right ear audio channel generated using the third processing technique 508 to obtain a single right ear audio channel including a reverb effect. Then, the virtualized reverb application 120 outputs a rendered stereo audio signal including a single left ear audio channel and a single right ear audio channel for playback by headphones or other speaker devices capable of playing back stereo audio.

[0063] exist Figure 5 In the example shown, the input audio signal is a monaural audio signal comprising a single audio channel. However, in some cases, the input audio signal comprises more than one audio channel. For example, the input audio signal may comprise two or more audio channels. Figure 7 Block diagram 700 is shown of a technique for processing N channels of audio signals utilizing a virtualized reverberation algorithm based on measuring binaural room impulse responses in accordance with one or more aspects of various embodiments.

[0064] As shown, Figure 7 The processing technique shown in is very similar in operation to Figure 5 However, although Figure 5 The processing technique shown in FIG. 5 includes a single filtering process 502 for separating low frequency components of a single input audio signal from mid and high frequency components of a single input audio signal, but Figure 7 The processing technique shown in 702 includes n filtering processes 702 - 1 to 702 -N for separating low-frequency components of N audio channels included in the input audio signal from mid- and high-frequency components of the N audio channels included in the input audio signal.

[0065] For example, the virtualized reverberation application 120 implements a first filtering process 702-1 to separate a low-frequency component of a first audio channel included in the input audio signal from a mid-high frequency component of the first audio channel included in the input audio signal. For example, the first filtering process 702-1 includes applying a low-pass filter to the first audio channel included in the input audio signal to obtain a low-frequency component of the first audio channel. The low-frequency component of the first audio channel included in the input audio signal includes a component in the first audio channel whose frequency is less than a crossover frequency. The first filtering process 702-1 also includes applying a high-pass filter to the first audio channel included in the input audio signal to obtain a mid-high frequency component of the first audio channel. The mid-high frequency component of the first audio channel included in the input audio signal includes a component in the first audio channel whose frequency is greater than a crossover frequency.

[0066] Similarly, the virtualized reverb application 120 further applies a corresponding filtering process 702 to each additional audio channel included in the input audio signal to separate the low-frequency components of the additional audio channels included in the input audio signal from the mid- and high-frequency components of the additional audio channels included in the input audio signal. For example, the virtualized reverb application 120 implements an N-th filtering process 702-N to separate the low-frequency components of the N-th audio channel included in the input audio signal from the mid- and high-frequency components of the N-th audio channel included in the input audio signal.

[0067] After separating the low-frequency components of the N audio channels included in the input audio signal from the mid- and high-frequency components of the N audio channels included in the input audio signal, the virtualized reverb application 120 uses the first processing technique 504 as described above to add reverb to the low-frequency components of the N audio channels included in the input audio signal. In addition, the virtualized reverb application 120 uses the second processing technique 506 and the third processing technique 508 as described above to add reverb to the mid- and high-frequency components of the N audio channels included in the input audio signal.

[0068] Figure 8 is a flow chart of method steps for processing an audio signal according to one or more aspects of various embodiments. Figures 1 to 7 The method steps are described using systems and examples, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the various embodiments.

[0069] As shown, the method 800 begins at step 802, where the virtualized reverberation application 120 obtains a measured BRIR of an acoustic space, such as the measured BRIR of the acoustic space 202. In some examples, the BRIR measurement application 118 obtains the BRIR by measuring the BRIR of the acoustic space 202 and storing the measured BRIR of the acoustic space 202 as BRIR data 122 in the system storage 112. In some examples, the virtualized reverberation application 120 receives the measured BRIR of the acoustic space 202 directly from the BRIR measurement application 118. In such examples, the BRIR measurement application 118 measures the BRIR of the acoustic space 202 and provides the measured BRIR of the acoustic space 202 to the virtualized reverberation application 120. In some examples, the BRIR measurement application 118 measures the BRIR of the acoustic space 202 using one or more known techniques, such as an MLS technique, an IRS technique, a time-stretched pulse technique, or a logarithmic sine swept technique.

[0070] At step 804, the virtualized reverb application 120 receives an input audio signal. In some examples, the input audio signal includes a single audio channel. In other examples, the input audio signal includes multiple audio channels. In one example, the input audio signal is a stereo audio signal including a left audio channel and a right audio channel. In other examples, the input audio signal includes 5.1.2 audio channels, 7.1.2 audio channels, or some other number of audio channels.

[0071] At step 806, the virtualized reverb application 120 separates the low frequency components of the input audio signal from the mid-high frequency components of the input audio signal. In some examples, the virtualized reverb application 120 applies a low pass filter to the input audio signal to obtain low frequency components in the input audio signal having frequencies less than a crossover frequency. In such examples, the virtualized reverb application 120 further applies a high pass filter to the input audio signal to obtain mid-high frequency components in the input audio signal having frequencies greater than the crossover frequency. In one non-limiting example, the crossover frequency is 900 Hz. In other examples, the crossover frequency may be greater than or less than 900 Hz.

[0072] At step 808, the virtualized reverb application 120 converts the low frequency components of the input audio signal into a plurality of converted low frequency audio channels. For example, the virtualized reverb application 120 uses the encoding and decoding process 510 to convert the low frequency components of the input audio signal into ( For example,The virtualized reverb application 120 may use a low-order Ambisonics encoder-decoder to encode the low-frequency components of the input audio signal into four low-frequency Ambisonics encoded audio channels, and decode the four low-frequency Ambisonics encoded audio channels into a plurality of low-frequency decoded audio channels. In other examples, the virtualized reverb application 120 may use a different type of low-order encoder-decoder to convert the low-frequency components of the input audio signal.

[0073] At step 810, the virtualized reverb application 120 reduces the respective gains of the plurality of converted low-frequency audio channels. In one non-limiting example, the virtualized reverb application 120 reduces the respective gains of the plurality of converted low-frequency audio channels to values ​​between -8 dB and -15 dB. In other examples, the virtualized reverb application 120 may reduce the respective gains of the plurality of converted low-frequency audio channels to have different gain values.

[0074] At step 812, the virtualized reverb application 120 convolves the plurality of converted low-frequency audio channels with the low-frequency component 400 of the measured BRIR of the acoustic space 202. For example, the virtualized reverb application 120 uses the low-frequency convolution process 514 to convolve the plurality of converted low-frequency audio channels with the low-frequency component 400 of the measured BRIR of the acoustic space 202. The result of convolving the plurality of converted low-frequency audio channels with the low-frequency component 400 of the measured BRIR is a rendered low-frequency left audio channel that includes a reverb effect and a rendered low-frequency right audio channel that includes a reverb effect.

[0075] At step 814, the virtualized reverberation application 120 divides the mid-high frequency component 300 of the measured BRIR of the acoustic space 202 into the direct sound portion 302 and the reflected sound portion 304. In some examples, the virtualized reverberation application 120 divides the mid-high frequency component 300 of the measured BRIR of the acoustic space 202 into the direct sound portion 302 and the reflected sound portion 304 in time. In such examples, the virtualized reverberation application 120 may divide the mid-high frequency component 300 of the measured BRIR into a first time window 306 occurring before time T1 and a second time window 308 occurring after time T1 to separate the direct sound portion 302 from the reflected sound portion 304. In one non-limiting example, the value of time T1 is 2 ms. In other examples, the value of time T1 may be less than or greater than 2 ms.

[0076] At step 816, the virtualized reverb application 120 converts the mid-high frequency components of the input audio signal into a plurality of converted mid-high frequency audio channels. For example, the virtualized reverb application 120 uses the encoding and decoding process 518 to convert the mid-high frequency components of the input audio signal into ( For example, In one non-limiting example, the virtualized reverb application 120 uses a low-order Ambisonics encoder-decoder to encode the mid-high frequency components of the input audio signal into four mid-high frequency Ambisonics encoded audio channels, and decode the four mid-high frequency Ambisonics encoded audio channels into a plurality of mid-high frequency decoded audio channels. In other examples, the virtualized reverb application 120 may use a different type of low-order encoder-decoder to convert the mid-high frequency components of the input audio signal.

[0077] At step 818, the virtualized reverb application 120 reduces the respective gains of the plurality of converted mid-high frequency audio channels. In one non-limiting example, the virtualized reverb application 120 reduces the respective gains of the plurality of converted mid-high frequency audio channels to a value between -8 dB and -15 dB. In other examples, the virtualized reverb application 120 may reduce the respective gains of the plurality of converted mid-high frequency audio channels to have different gain values.

[0078] At step 820, the virtualized reverb application 120 convolves the plurality of converted mid-high frequency audio channels with the reflected sound portion 304 included in the mid-high frequency component 300 of the measured BRIR of the acoustic space 202. For example, the virtualized reverb application 120 uses the reflection convolution process 522 to convolve the plurality of converted mid-high frequency audio channels with the reflected sound portion 304 included in the mid-high frequency component 300 of the measured BRIR of the acoustic space 202. The result of convolving the plurality of converted mid-high frequency audio channels with the reflected sound portion 304 included in the mid-high frequency component 300 of the measured BRIR is a rendered mid-high frequency left audio channel including a reverberation effect associated with the reflected sound portion of the measured BRIR and a rendered mid-high frequency right ear audio channel including a reverberation effect associated with the reflected sound portion of the measured BRIR.

[0079] At step 822, the virtualized reverb application 120 convolves the discrete mid-high frequency components of the input audio signal with the direct sound portion 302 included in the mid-high frequency components 300 of the measured BRIR of the acoustic space 202. That is, the virtualized reverb application 120 does not convert the mid-high frequency components of the input audio signal before convolving the mid-high frequency components of the input audio signal with the direct sound portion 302 included in the mid-high frequency components 300 of the measured BRIR of the acoustic space 202. For example, the virtualized reverb application 120 uses the direct convolution process 526 to convolve the discrete mid-high frequency components of the input audio signal with the direct sound portion 302 included in the mid-high frequency components 300 of the measured BRIR of the acoustic space 202. The result of convolving the discrete mid-high frequency audio channels with the direct sound portion 302 included in the mid-high frequency component 300 of the measured BRIR is a rendered mid-high frequency left audio channel including the reverberation effect associated with the direct sound portion of the measured BRIR and a rendered mid-high frequency right audio channel including the reverberation effect associated with the direct sound portion of the measured BRIR.

[0080] At step 824, the virtualized reverb application 120 equalizes the rendered left audio channel and the rendered right audio channel. For example, the virtualized reverb application 120 applies the headphone equalization process 516 to the rendered low-frequency left audio channel and the rendered low-frequency right audio channel including the reverb effect. As another example, the virtualized reverb application 120 applies the headphone equalization process 524 to the rendered high-mid-range left audio channel and the rendered high-mid-range right audio channel including the reverb effect associated with the reflected sound portion of the measured BRIR. As another example, the virtualized reverb application 120 applies the headphone equalization process 528 to the rendered high-mid-range left audio channel and the rendered high-mid-range right audio channel including the reverb effect associated with the direct sound portion of the measured BRIR.

[0081] At step 826, the virtualized reverb application 120 sums or combines the rendered left audio channel and sums or combines the rendered right audio channel. For example, the virtualized reverb application 120 implements the summing process 530 to combine the rendered low-frequency left audio channel including the reverb effect, the rendered mid-high frequency left audio channel including the reverb effect associated with the reflected sound portion of the measured BRIR, and the rendered mid-high frequency left audio channel including the reverb effect associated with the direct sound portion of the measured BRIR into a single rendered left audio channel including the reverb effect. Similarly, the virtualized reverb application 120 implements the summing process 530 to combine the rendered low-frequency right audio channel including the reverb effect, the rendered mid-high frequency right audio channel including the reverb effect associated with the reflected sound portion of the measured BRIR, and the rendered mid-high frequency right audio channel including the reverb effect associated with the direct sound portion of the measured BRIR into a single rendered right audio channel including the reverb effect. At step 828 , the virtualized reverb application 120 outputs a rendered stereo signal including a single rendered left audio channel including the reverb effect and a single rendered right audio channel including the reverb effect.

[0082] In summary, a computing device renders an audio signal with reverberation based on a measured binaural room impulse response (BRIR) of an acoustic space to add the reverberation effect of the acoustic space for playback using headphones. Using the disclosed technology, the computing device uses three separate processing techniques to add reverberation to the audio signal. In addition, using the disclosed technology, the computing device separates the audio signal into a low-frequency component and a high-frequency component. Using the first processing technique, the computing device converts the low-frequency component of the audio signal into a first plurality of converted audio channels, and convolves the first plurality of converted audio channels with the measured BRIR to generate a first left audio channel and a first right audio channel. Using the disclosed technology, the computing device further divides the measured BRIR into a first time window including a direct sound portion of the measured BRIR and a second time window including a reflected sound portion of the measured BRIR. Using the second processing technique, the computing device converts the mid- and high-frequency components of the audio signal into a second plurality of converted audio channels, and convolves the second plurality of converted audio channels with the reflected sound portion of the measured BRIR to generate a second left audio channel and a second right audio channel. Using the third processing technique, the computing device convolves the discrete high frequency components of the audio signal with the direct sound portion of the measured BRIR to generate a third left audio channel and a third right audio channel. The direct sound portion of the measured BRIR corresponds to the direction in which the sound emitted by the source being rendered directly reaches the listener's ear. Using the disclosed technology, the computing device combines the results of the first processing technique, the second processing technique, and the third processing technique to generate left and right ear sounds including reverberation. For example, the computing device combines the first left audio channel, the second left audio channel, and the third left audio channel into a single left audio channel, and combines the first right audio channel, the second right audio channel, and the third right audio channel into a single right audio channel. Then, the computing device outputs an output audio signal including the combined left audio channel and the combined right audio channel for playback by headphones. Before or after combining the left and right ear audio channels, the computing device may optionally apply headphone equalization to the left and right audio channels.

[0083] At least one technical advantage of the disclosed techniques over the prior art is that, using the disclosed techniques, reverberation can be added to an audio signal at a relatively low computational cost using a measured room impulse response of an acoustic space. Thus, using the disclosed techniques, relatively modest processing power can be used to render reverberation for a large number of sound sources and acoustic spaces based on a measured room impulse response that sounds more natural than a room impulse response derived from an acoustic space model. These technical advantages provide one or more technical improvements over prior art methods.

[0084] 1. In some embodiments, a computer-implemented method for audio processing includes: obtaining a binaural room impulse response (BRIR) of an acoustic space; receiving an input audio signal; separating the input audio signal into a low-frequency component and a high-frequency component; dividing the BRIR of the acoustic space into a first part occurring before a first time and a second part occurring after the first time; generating a first component of an output audio signal based on the high-frequency component of the input audio signal and the first part of the BRIR; generating a second component of the output audio signal based on the high-frequency component of the input audio signal and the second part of the BRIR; generating a third component of the output audio signal based on the low-frequency component of the input audio signal and the BRIR; and outputting the output audio signal.

[0085] 2. The computer-implemented method of clause 1, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

[0086] 3. A computer-implemented method as described in clause 1 or clause 2, wherein generating the first component of the output audio signal includes convolving the high frequency component of the input audio signal with the first portion of the BRIR to generate a first left audio channel and a first right audio channel.

[0087] 4. The computer-implemented method of any of clauses 1 to 3, further comprising applying headphone equalization to the first left audio channel and the first right audio channel.

[0088] 5. A computer-implemented method as described in any of clauses 1 to 4, wherein the first part of the BRIR corresponds to the direction from which sounds emitted by sources in the acoustic space directly reach ears of listeners.

[0089] 6. A computer-implemented method as described in any of clauses 1 to 5, wherein generating the second component of the output audio signal comprises: converting the high frequency component of the input audio signal into a first plurality of converted audio channels; and convolving the first plurality of converted audio channels with the second portion of the BRIR to generate a second left audio channel and a second right audio channel.

[0090] 7. The computer-implemented method of any of clauses 1 to 6, wherein converting the high frequency components of the input audio signal into the first plurality of converted audio channels comprises using a first order Ambisonics encoder-decoder.

[0091] 8. The computer-implemented method of any of clauses 1 to 7, further comprising reducing a gain of the first plurality of converted audio channels prior to convolving the first plurality of converted audio channels with the second portion of the BRIR.

[0092] 9. A computer-implemented method as described in any of clauses 1 to 8, wherein generating the third component of the output audio signal comprises: converting the low-frequency component of the input audio signal into a second plurality of converted audio channels; and convolving the second plurality of converted audio channels with the BRIR to generate a third left audio channel and a third right audio channel.

[0093] 10. The computer-implemented method of any of clauses 1 to 9, wherein obtaining the BRIR of the acoustic space comprises measuring the BRIR of the acoustic space using a microphone.

[0094] 11. The computer-implemented method of any of clauses 1 to 10, wherein the output audio signal comprises a left channel and a right channel.

[0095] 12. According to some embodiments, one or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors at a computing device, cause the one or more processors to perform the following steps: obtain a binaural room impulse response (BRIR) of an acoustic space; receive an input audio signal; separate the input audio signal into a low-frequency component and a high-frequency component; divide the BRIR of the acoustic space into a first part occurring before a first time and a second part occurring after the first time; generate a first component of an output audio signal based on the high-frequency component of the input audio signal and the first part of the BRIR; generate a second component of the output audio signal based on the high-frequency component of the input audio signal and the second part of the BRIR; generate a third component of the output audio signal based on the low-frequency component of the input audio signal and the BRIR; and output the output audio signal.

[0096] 13. The one or more non-transitory computer-readable media of clause 12, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

[0097] 14. One or more non-transitory computer-readable media as described in clause 12 or clause 13, wherein generating the first component of the output audio signal includes convolving the high frequency component of the input audio signal with the first portion of the BRIR to generate a first left audio channel and a first right audio channel; generating the second component of the output audio signal includes: converting the high frequency component of the input audio signal into a first plurality of converted audio channels, and convolving the first plurality of converted audio channels with the second portion of the BRIR to generate a second left audio channel and a second right audio channel; and generating the third component of the output audio signal includes: converting the low frequency component of the input audio signal into a second plurality of converted audio channels, and convolving the second plurality of converted audio channels with the BRIR to generate a third left audio channel and a third right audio channel.

[0098] 15. One or more non-transitory computer-readable media as described in any of clauses 12 to 14, wherein the steps further include: applying headphone equalization to the first left audio channel and the first right audio channel; applying headphone equalization to the second left audio channel and the second right audio channel; and applying headphone equalization to the third left audio channel and the third right audio channel.

[0099] 16. One or more non-transitory computer-readable media as described in any of clauses 12 to 15, wherein the steps further include: reducing the gain of the first plurality of converted audio channels before convolving the first plurality of converted audio channels with the second portion of the BRIR; and reducing the gain of the second plurality of converted audio channels before convolving the second plurality of converted audio channels with the BRIR.

[0100] 17. One or more non-transitory computer-readable media as described in any of clauses 12 to 16, wherein converting the high-frequency components of the input audio signal into the first plurality of converted audio channels comprises using a first-order Ambisonics encoder-decoder; and converting the low-frequency components of the input audio signal into the second plurality of converted audio channels comprises using a first-order Ambisonics encoder-decoder.

[0101] 18. The one or more non-transitory computer-readable media of any of clauses 12 to 17, wherein obtaining the BRIR of the acoustic space comprises measuring the BRIR of the acoustic space using a microphone.

[0102] 19. According to some embodiments, a computing device comprising: a memory storing an application; and one or more processors, which are configured to obtain a binaural room impulse response (BRIR) of an acoustic space when executing the application; receive an input audio signal; separate the input audio signal into a low-frequency component and a high-frequency component; divide the BRIR of the acoustic space into a first part occurring before a first time and a second part occurring after the first time; generate a first component of an output audio signal based on the high-frequency component of the input audio signal and the first part of the BRIR; generate a second component of the output audio signal based on the high-frequency component of the input audio signal and the second part of the BRIR; generate a third component of the output audio signal based on the low-frequency component of the input audio signal and the BRIR; and output the output audio signal.

[0103] 20. The computing device of clause 19, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

[0104] Any and all combinations of any claim elements recited in any claim and / or any elements described in the present application, in any manner, are within the intended scope of the present disclosure and protection.

[0105] The description of the various embodiments has been presented for purposes of illustration and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0106] Aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the following forms: a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.) or an implementation combining software aspects and hardware aspects, which may generally be referred to as a "module", "system" or "computer" in this article. In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or a collection of circuits. In addition, aspects of the present disclosure may take the form of a computer program product, which is embodied in one or more computer-readable media on which a computer-readable program code is embodied.

[0107] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following media: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing media. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device, or apparatus.

[0108] Aspects of the present disclosure are described with reference to the flowchart illustrations and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart illustration and / or block diagram and the frame combination of the flowchart illustration and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. When the processor of a computer or other programmable data processing device executes instructions, it is possible to realize the function / action specified in one or more frames of the flowchart and / or block diagram. Such processors may be, but are not limited to, general-purpose processors, special-purpose processors, special-purpose processors or field programmable gate arrays.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, section or portion of a code, and the code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the box may not appear in the order mentioned in the accompanying drawings. For example, depending on the functions involved, the two boxes shown in succession may be executed substantially simultaneously, or the boxes may sometimes be executed in the opposite order. It should also be noted that each box of the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration may be implemented by a dedicated hardware-based system that performs a specified function or action or by a combination of dedicated hardware and computer instructions.

[0110] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, and the scope of the present disclosure is determined by the claims that follow.

Claims

1. A computer-implemented method for audio processing, the method include: Obtain the binaural room impulse response (BRIR) of the acoustic space; receiving an input audio signal; Separating the input audio signal into a low-frequency component and a high-frequency component; dividing the BRIR of the acoustic space into a first portion occurring before a first time and a second portion occurring after the first time; generating a first component of an output audio signal based on the high frequency component of the input audio signal and the first portion of the BRIR; generating a second component of the output audio signal based on the high frequency component of the input audio signal and the second portion of the BRIR; generating a third component of the output audio signal based on the low frequency component of the input audio signal and the BRIR; as well as The output audio signal is output.

2. The computer-implemented method of claim 1, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

3. The computer-implemented method of claim 1 , wherein generating the first component of the output audio signal comprises convolving the high frequency component of the input audio signal with the first portion of the BRIR to generate a first left audio channel and a first right audio channel.

4. The computer-implemented method of claim 3, further comprising applying headphone equalization to the first left audio channel and the first right audio channel.

5. The computer-implemented method of claim 1, wherein the first portion of the BRIR corresponds to a direction in which sounds emitted by sources in the acoustic space directly reach ears of a listener.

6. The computer-implemented method of claim 1 , wherein the second component of the output audio signal is generated include: converting the high frequency component of the input audio signal into a first plurality of converted audio channels; as well as The first plurality of transformed audio channels are convolved with the second portion of the BRIR to generate a second left audio channel and a second right audio channel.

7. The computer-implemented method of claim 6, wherein converting the high frequency components of the input audio signal into the first plurality of converted audio channels comprises using a first order Ambisonics encoder-decoder.

8. The computer-implemented method of claim 6, further comprising: include: The gains of the first plurality of converted audio channels are reduced before convolving the first plurality of converted audio channels with the second portion of the BRIR.

9. The computer-implemented method of claim 1 , wherein the third component of the output audio signal is generated include: converting the low frequency components of the input audio signal into a second plurality of converted audio channels; as well as The second plurality of transformed audio channels are convolved with the BRIR to generate a third left audio channel and a third right audio channel.

10. The computer-implemented method of claim 1, wherein obtaining the BRIR of the acoustic space comprises measuring the BRIR of the acoustic space using a microphone.

11. The computer-implemented method of claim 1 , wherein the output audio signal comprises a left channel and a right channel.

12. One or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors at a computing device, cause the one or more processors to perform the following steps: Obtain the binaural room impulse response (BRIR) of the acoustic space; receiving an input audio signal; Separating the input audio signal into a low-frequency component and a high-frequency component; dividing the BRIR of the acoustic space into a first portion occurring before a first time and a second portion occurring after the first time; generating a first component of an output audio signal based on the high frequency component of the input audio signal and the first portion of the BRIR; generating a second component of the output audio signal based on the high frequency component of the input audio signal and the second portion of the BRIR; generating a third component of the output audio signal based on the low frequency component of the input audio signal and the BRIR; as well as The output audio signal is output.

13. The one or more non-transitory computer-readable storage media of claim 12, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

14. One or more non-transitory computer-readable storage media as claimed in claim 12, in: Generating the first component of the output audio signal comprises convolving the high frequency component of the input audio signal with the first portion of the BRIR to generate a first left audio channel and a first right audio channel; Generating the second component of the output audio signal comprises: converting the high frequency components of the input audio signal into a first plurality of converted audio channels; and convolving the first plurality of converted audio channels with the second portion of the BRIR to generate a second left audio channel and a second right audio channel; and Generating the third component of the output audio signal comprises: converting the low frequency components of the input audio signal into a second plurality of converted audio channels; and The second plurality of transformed audio channels are convolved with the BRIR to generate a third left audio channel and a third right audio channel.

15. One or more non-transitory computer-readable storage media as described in claim 14, wherein the step further include: applying headphone equalization to the first left audio channel and the first right audio channel; applying headphone equalization to the second left audio channel and the second right audio channel; as well as Headphone equalization is applied to the third left audio channel and the third right audio channel.

16. One or more non-transitory computer-readable storage media as described in claim 14, wherein the step further include: reducing a gain of the first plurality of converted audio channels before convolving the first plurality of converted audio channels with the second portion of the BRIR; as well as The gains of the second plurality of converted audio channels are reduced before convolving the second plurality of converted audio channels with the BRIR.

17. One or more non-transitory computer-readable storage media as claimed in claim 14, in: Converting the high frequency components of the input audio signal into the first plurality of converted audio channels comprises using a first order Ambisonics encoder-decoder; as well as Converting the low frequency components of the input audio signal into the second plurality of converted audio channels comprises using a first order Ambisonics encoder-decoder.

18. The one or more non-transitory computer-readable storage media of claim 12, wherein obtaining the BRIR of the acoustic space comprises measuring the BRIR of the acoustic space using a microphone.

19. A computing device, include: a memory storing an application program; as well as One or more processors, wherein the one or more processors are configured to: Obtain the binaural room impulse response (BRIR) of the acoustic space; receiving an input audio signal; Separating the input audio signal into a low-frequency component and a high-frequency component; dividing the BRIR of the acoustic space into a first portion occurring before a first time and a second portion occurring after the first time; generating a first component of an output audio signal based on the high frequency component of the input audio signal and the first portion of the BRIR; generating a second component of the output audio signal based on the high frequency component of the input audio signal and the second portion of the BRIR; generating a third component of the output audio signal based on the low frequency component of the input audio signal and the BRIR; as well as The output audio signal is output.

20. The computing device of claim 19, wherein the first portion of the BRIR comprises a direct sound portion and the second portion of the BRIR comprises a reflected sound portion.

Images