Devices, methods, and computer-readable media providing binaural virtualization

Abstract

Systems and methods are provided for binaural virtualization that further simplify the process by mode switching between two different filtering modes: a standard mode and a simplified mode.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 266,500, filed January 6, 2022; U.S. Provisional Application No. 63 / 168,340, filed March 31, 2021, entitled “LIGHTWEIGHT VIRTUALIZER FOR BINAURAL SIGNAL GENERATION FROM STEREO”; and International Application No. PCT / CN 2021 / 077922, filed February 25, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to improvements in binaural processing. More specifically, it relates to methods and systems for providing lightweight processes for binaural processing. Background Technology

[0004] Audio systems typically consist of audio sources (such as radio receivers, smartphones, laptops, desktop computers, tablets, televisions, etc.) and speakers. In some cases, the speakers are worn near the listener's ears, such as headphone and earbuds. In these situations, it is sometimes necessary to simulate the audio quality of external speakers that are not close to the ears. This can be accomplished by synthesizing sound to create a binaural effect before sending the audio to the near-end speakers (hereinafter referred to as headphones).

[0005] The methods described in this section are permissible methods, but not necessarily methods previously conceived or employed. Therefore, unless otherwise stated, no method described in this section should be assumed to conform to the prior art solely by virtue of its inclusion in this section. Similarly, unless otherwise stated, issues relating to one or more methods should not be assumed to have been recognized in any prior art based on this section. Summary of the Invention

[0006] While sound is synthesized before being sent to speakers to create a binaural effect, not all audio sources are configured to perform this synthesis, and normal synthesis circuitry is too memory-intensive and complex to be included in headphones or earbuds.

[0007] The methods and systems / devices described in this paper provide a low-complexity (lightweight) apparatus for creating high-quality binaural effects with channel-level controlled reverberation. Furthermore, this allows for binaural virtualization in small devices, including headphones and earbuds, which is typically not feasible.

[0008] This publication describes systems and methods for providing lightweight binaural virtualization, which can be included in headphones, earbuds, or other memory- and complexity-sensitive devices. The systems and methods can be implemented as part of an audio decoder.

[0009] One embodiment of the present invention is a device for providing binaural virtualization, the device comprising: inputs of a left input signal and a right input signal; a virtualizer; an upmixer configured to convert the left input signal and the right input signal into a right channel, a left channel, and a center channel; a mixer configured to combine the left input signal with the left channel based on a center-only reverb amount value, and to combine the right input signal with the right channel based on a center-only reverb amount value, thereby producing a mixer output; and a reverb module configured to apply reverb response to the mixer output for the virtualizer.

[0010] One embodiment of the present invention is a method for providing binaural virtualization, the method comprising: receiving inputs of a left input signal and a right input signal; upmixing the left input signal and the right input signal into a right channel, a left channel, and a center channel; mixing the left input signal with the left channel based on a center-only reverb value, and mixing the right input signal with the right channel based on a center-only reverb value, thereby generating a mixer output; and applying the mixing response to the mixer output for the virtualizer.

[0011] These embodiments are exemplary and not limiting: other embodiments may be contemplated based on the disclosure herein. Attached Figure Description

[0012] Figure 1 An example of the use of a lightweight virtualizer is shown.

[0013] Figure 2 An example of binaural audio is shown.

[0014] Figure 3 An example setup for a lightweight virtualizer is shown.

[0015] Figure 4 An example reverb control for a lightweight virtualizer is shown.

[0016] Figures 5A-5B An example of a lightweight virtualizer setup is shown. Figure 5A A simple virtual machine is shown, and Figure 5B A more efficient virtualizer was demonstrated.

[0017] Figures 6A-6B An example of a reverb generation mode is shown. Figure 6A The full mode is shown, and Figure 6B A simplified mode is shown.

[0018] Figure 7 An example upmixer procedure for a lightweight virtualizer is shown.

[0019] Figure 8 An example of a lightweight virtualizer method is shown. Detailed Implementation

[0020] As used in this article, "lightweight" refers to a circuit implementation that reduces memory and complexity. This reduces the circuit's footprint and power consumption.

[0021] As used in this article, "HRIR" refers to head-related impulse response. This can be considered a time-domain representation of the HRTF (head-related transfer function), which describes how the ear receives sound from a source.

[0022] As used in this article, "ITD" refers to interaural time difference, which describes the time difference between each ear receiving a given instance of sound from a source.

[0023] As used in this article, "ILD" refers to interaural level difference, which describes the difference in perceived amplitude received by each ear from a given instance of sound from a source.

[0024] As used in this article, a “Butterworth filter” refers to a filter that is essentially flat in the passband.

[0025] As used in this article, "binaural" refers to sound delivered separately to each ear, with multiple speakers placed at a distance from the listener and spaced apart from each other.

[0026] As used in this article, "virtual device" refers to a system capable of synthesizing binaural sound.

[0027] As used in this article, "upmixing" is the process of converting M input channels into N output channels, where N > M (an integer). An "upmixer" is a module that performs upmixing.

[0028] As used herein, a "signal" is an electronic representation of audio or video input or output from a system. A signal can be stereo (left and right signals separated). As used herein, a "channel" is a portion of the signal processed by the system. Examples of channels include left, right, and center.

[0029] As used in this article, a "module" refers to a part of hardware, software, or firmware that performs a specific function. Modules are not necessarily physically separated from each other in implementation.

[0030] As used in this article, “input stage” refers to the hardware and / or software / firmware that processes the reception of input signals for a device.

[0031] Figure 1 An example use of the lightweight virtualizer is shown. The user has a mobile device (105), such as a smartphone or tablet, which is connected to a stereo listening device (110), such as earbuds, wired or wireless headphones, or a portable speaker. If the sound-providing application (“application”) running on the mobile device (105) does not provide binaural sound, the listening device (110) with the lightweight virtualizer can synthesize a binaural effect.

[0032] Figure 2 An example of binaural sound is shown. In the non-synthesized system, two speakers (205) are placed on the left and right sides in front of the listener. This arrangement results in a non-zero ITD and ILD, or "crosstalk," between the path (210) from each speaker to the closer listener's ear (220) and the path (215) to the opposite ear (220). Virtualization attempts to synthesize this effect for the headphones (220).

[0033] From C. Phillip Brown, “ A Structural Model for Binaural Sound Synthesis [Structural Model of Binaural Sound Synthesis] The HRIR head model, as described in the IEEE Transactions on Speech and Audio Processing, Vol. 6, No. 5, September 1998, is a combination of ITD and ILD. Based on the formula by Woodworth and Schlosberg (see Woodworth, RS and Schlosberg, H. (1962), Experimental Psychology (Holt, New York), pp. 348–361), the ITD model is related to the head radius and angle. When the elevation angle is set to zero, the formula becomes:

[0034] The ILD signal can be approximated by adding a minimum phase filter to account for the amplitude response (head shadow). The ILD filter can also provide the observed frequency-dependent delay.

[0035]

[0036] By cascading ITD and ILD, the time-domain filter is:

[0037] A harmonic generator can primarily produce harmonics based on the center channel. This harmonic generator is designed to provide a virtual bass effect. It uses multiplication of each sample itself to generate harmonics.

[0038]

[0039] The equalizer can apply parametric filters or shelving filters, as used in SO. J. Orfanidis, " High-Order Digital Parametric Equalizer Design The method of J. Audio Eng. Soc., Vol. 53, No. 11, pp. 1026-1046 (November 2005).

[0040] Figure 3 An example of a basic lightweight virtualizer layout is shown. The input (305), consisting of left and right input signals, is sent to a reverb module before upmixing (310) to generate left and right reverb for the virtualizer module (390), and then to an upmixer module (315) to convert the left and right input signals into left, right, and center channels. The left, right, and center channels can then be sent to a harmonic generator (320) and an equalizer (325) to improve sound quality. The virtualizer module (390) synthesizes the reverb output and the left, right, and center channels into the binaural output (395) of the headphones.

[0041] In some embodiments, binaural sound is synthesized by controlling the amount of reverberation in the channels by adjusting the amplitude based on the total reverberation amount value.

[0042] Figure 4 An example of reverb control is shown. Before being processed by the virtualizer (400), the left and right input signals (405) and the left and right reverb channels (410) are combined by the mixer (412). They are adjusted by a total reverb value (reverb_amount), which ranges between no reverb (0 in this example) and full reverb (1 in this example). The mix is ​​proportional to the total reverb value. The mix can be represented as:

[0043] in It is the total reverberation value. It is the reverberation signal input (L) rev and R rev ), This is the raw input (L and R channels). The reverberation can be smoothed block by block using a first-order smoothing filter to avoid glitches caused by changes in reverberation.

[0044] The mixer output (413) is then passed through the ipsi (415-I) and contra (415-C) filters and then mixed with the center channel (420) to create a virtualized binaural signal output (425).

[0045] Control over the total reverb level allows for control over the virtualization, enabling headphone manufacturers to adapt the virtualization to the specific hardware of the headphones and / or the user to tailor the virtualization experience. In some embodiments, only the center reverb level can be controlled by an API (application programming interface), such as an application on the device paired with the headphones. This control can be automated by the software of the mobile device (e.g., when a sound in the audio that should have reverb reduced is detected), or it can be set and / or adjusted by the user through a user interface to provide a customized virtualization experience. In some embodiments, only the center reverb level is set or adjusted by the headphones themselves (e.g., a preset or offset value in the software / firmware) to provide an optimal balance based on how the hardware handles reverb.

[0046] In some embodiments, only the center reverb level is controlled independently of the total reverb level (given options with values ​​that differ from each other). This helps to control the center-vs-(left+right) reverb level, for example, to avoid excessive reverb on the voice audio in the center channel while still providing enough reverb for the music to deliver a virtualized 3D experience.

[0047] A simple method to generate reverb on the center channel is as follows: Figure 5A As shown. The reverb module (505) is fed from the upmixer (510) along with the center channel of the left and right channels. As shown in this example, a limiter (515) can be used to prevent clipping from exceeding the digital range.

[0048] More effective methods for generating reverberation on the center channel include... Figure 5B As shown. The reverb module (555) is instead fed a mixed input from the input channel (565) and the upmixed left and right channels (570) from the upmixer (560). The mix is ​​controlled by only the center reverb value (center_reverb_amount), similar to Figure 4 The mix shown. The L and R input signals have a center_reverb_amount applied to them. (See gain module 575), while the upmixed L and R channels have an additive inverse of center_reverb_amount relative to 1 applied to them. (See gain module 576). The effect is that when the center-only reverb value is at its maximum (e.g., 1), then the center channel will have full reverb (the reverb module (555) will only receive the pre-mixed left and right input signals, which inherently include the center channel). When the center-only reverb value is no reverb (e.g., 0), then the center channel will have no reverb (the reverb module (555) will only receive the post-mixed left and right channels, where the center channel has been removed). Intermediate values ​​will proportionally adjust the center-only reverb (e.g., 0.5 will treat a half-reverb center channel as a left or right channel). The left and right reverb amounts remain constant depending on the center-only reverb value - they are controlled only by the total reverb setting.

[0049] Both the center reverb value and the total reverb value can be controlled separately by the API.

[0050] Efficient reverberation generation methods (e.g., Figure 5B ) is more complex than simpler systems (e.g., Figure 5A This saves memory usage and complexity, which is an important step in making the system more lightweight, as reverb generators typically contribute a large portion of the system's memory usage and complexity.

[0051] In some embodiments, the mixing ratio is controlled as a piecewise non-linear function, for example:

[0052] in This is only the center reverb value (e.g., API setting). It is a constant used to standardize the results (providing consistent volume). This is a value from the upmixer, which gives the proportion of the left or right channel (e.g., the left channel) in the center channel. It is a threshold. ( ) represents the applied center-only reverb level. This helps avoid asymmetrical audio content in the left and right channels.

[0053] In some embodiments, reverb generation can switch between two complexity modes.

[0054] Figure 6A and 6B An example of providing variable complexity for reverb generation is shown.

[0055] Figure 6AThe normal (full complexity) operating mode is shown. Here, the reverberation generator works in conjunction with a low-pass (e.g., Butterworth) filter (605), which is fed into a comb filter (610) and then into a full-pass filter (615) to change the phase. The comb filter (610) consists of multiple infinite impulse response (IIR) filters with different delay values. This is memory and complexity intensive and may produce a stronger reverberation than expected.

[0056] The Z-domain expressions for comb filters and all-pass filters are as follows:

[0057] in and It is the reflection gain, and It is the delay in the sample.

[0058] Figure 6B The simplified mode is shown, where a low-pass filter (655) is directly fed into an all-pass filter (660) with a longer phase delay (to simulate a large room) and a stronger reflectivity. The audio volume is also increased to compensate, resulting in generally clearer sound for audio with less reverberation. Compared to the normal mode, the simplified mode reduces memory usage and complexity, thus the ability to switch modes when needed (e.g., in situations where memory and complexity are critical) helps the lightweight virtualizer operate in a variety of conditions.

[0059] The following description of further embodiments will focus on their differences from the embodiments described above. Therefore, features common to both embodiments will be omitted from the following description, and thus, unless otherwise required by the following description, it should be assumed that features of the previously described embodiments are, or at least can be, implemented in the further embodiments. In some embodiments, the lightweight virtualizer can detect whether virtualization is unnecessary and bypass it. This can be achieved through API instructions, machine learning-derived binaural detection (see, for example, Zhang Chunmao et al.'s "..."). Blind Detection Of Binauralized Stereo Content [Blind testing of binaural stereo content] ", WO2019 / 209930A1, the full text of which is incorporated herein by reference), or by receiving an identification of a known virtualized mobile device or mobile device application.

[0060] Figure 7 An example of an upmixer (2-channel to 3-channel upmixing) is shown. It derives a virtual center channel from the left and right channels, thereby achieving decorrelation of the left and right channels and enhancing the separability of the binaural signals. The upmixing process is a form of feedback-free active matrix decoding (see, for example, C. Phillip Brown, "..."). Method and System for Frequency Domain Active Matrix Decoding without Feedback [with system] (WO 2010 / 083137 A1, incorporated herein by reference in its entirety). The upmixer treats the sum of the left and right channels as the center channel (C) and the difference between them as the side channels (S). The power (P) of the four channels can be calculated and smoothed. The power ratios of the left (L), right (R), front (F), and rear (B) channels can be derived from the power. The upmix coefficients for the left, right, front, and rear channels are calculated based on nonlinear power ratios. The derived virtual center channel is a weighted linear combination of the left and right channels. In this example, the channels are summed and differencing (705) to provide the left, right, center, and side channels. The power sum and power difference (710) give the power level that is then smoothed (715). The power ratios for the left, right, front, and rear channels are derived (720), the upmix coefficients (725) are calculated, and the center channel is derived (730).

[0061] Figure 8 A flowchart example of a basic lightweight virtualizer method is shown. The system receives left and right input signals from an audio source at the input stage (805). These signals are then upmixed (810) to upmixed versions for the left, right, and center channels. The upmixed left and right channels and the input signals are then mixed (815) based on a scaling factor (center reverb only, set by the system or API (830)). Reverb is then provided to the mixed channels based on a total reverb amount (820), also set by the system or API (840). The outputs are then presented as left and right reverb channel outputs (835) for further processing (e.g., virtualization using input or post-processing inputs).

[0062] Several embodiments of this disclosure have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims.

[0063] The examples described above are provided to those skilled in the art as a complete disclosure and description of how to make and use the disclosed embodiments, and are not intended to limit the scope of the disclosure as believed by the inventors.

[0064] Modifications made to the above-described models for performing the methods and systems disclosed herein, which are obvious to those skilled in the art, are intended to fall within the scope of the following claims. All patents and publications mentioned in the specification demonstrate the level of expertise of those skilled in the art in relation to this disclosure.

[0065] It should be understood that this disclosure is not limited to any particular method or system, which may of course vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, unless expressly specified otherwise, the singular forms “a,” “an,” and “the” include plural references. Unless expressly specified otherwise, the term “a plurality” includes two or more referents. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

Claims

1. A device for providing binaural virtualization, the device comprising: An input stage configured to receive a left input signal and a right input signal; An upmixer configured to convert the left input signal and the right input signal into a right channel, a left channel, and a center channel; A mixer configured to combine the left input signal with the left channel based on a center-only reverb value, and to combine the right input signal with the right channel based on the center-only reverb value, thereby producing a mixer output; A reverb module configured to apply a reverb response to the mixer output and generate left and right reverb channels; as well as A virtualizer configured to receive the left channel, the right channel, the center channel, and the left and right reverb channels as inputs, and to output binaural signals.

2. The device as claimed in claim 1, wherein, The reverberation module is configured to adjust the reverberation using the total reverberation value.

3. The device as described in claim 2, wherein, The center reverberation value and the total reverberation value are set independently.

4. The device as claimed in any one of claims 1 to 3, further comprising at least one of a harmonic generator and an equalizer between the upmixer and the virtualizer.

5. The device as described in any one of claims 1 to 3, wherein, The device is configured to detect whether the left input signal and the right input signal are already binaural.

6. The device as claimed in claim 5, wherein, The device detects whether the left and right input signals are already binaural by receiving identifiers of the sources of the left and right input signals.

7. The device as claimed in claim 5, wherein, The device uses machine learning to detect whether the left and right input signals are already binaural.

8. The device as claimed in claim 5, wherein, The device detects whether the left input signal and the right input signal are already binaural via application programming interface instructions.

9. The device as claimed in any one of claims 1 to 3, wherein, The virtual device is part of the audio decoder.

10. A method for providing binaural virtualization, the method comprising: It receives input signals from the left and right input signals; The left input signal and the right input signal are upmixed to form the right channel, left channel and center channel; The left input signal is mixed with the left channel based on the center reverb value only, and the right input signal is mixed with the right channel based on the center reverb value only, thereby producing a mixer output; The reverb response is applied to the mixer output to produce left and right reverb channels; The left channel, the right channel, the center channel, and the left and right reverb channels are input to the virtualizer; as well as The virtual device outputs binaural signals.

11. The method of claim 10, further comprising adjusting the reverberation by a total reverberation value.

12. The method of claim 11, wherein, The center reverberation value and the total reverberation value are set by the application programming interface.

13. The method of any one of claims 10 to 12, further comprising at least one of harmonic generation and equalization after the upmixing.

14. The method of any one of claims 10 to 12, further comprising detecting whether the left input signal and the right input signal are already binaural.

15. The method of claim 14, wherein, The detection is performed by receiving identifiers from the sources of the left input signal and the right input signal.

16. The method of claim 14, wherein, The detection was performed using machine learning-based binaural detection.

17. The method of claim 14, wherein, The detection is performed via application programming interface (API) instructions.

18. The method of any one of claims 10 to 12, further comprising switching between a standard filtering mode and a simplified filtering mode, wherein, The standard filtering mode includes the use of a comb filter, while the simplified filtering mode does not include the use of the comb filter.

19. A non-transitory computer-readable medium comprising data having a computer program code portion configured to perform the steps of the method as claimed in any one of claims 10 to 18 when executed by a processor.

Images