3D audio source spatialization

Abstract

Techniques for delivering audio in a telepresence system include specifying a frequency threshold below which crosstalk cancellation (CC) is used and above which VBAP is used. In some embodiments, such a frequency threshold is between 1000 Hz and 2000 Hz. In addition, in some embodiments, improved techniques include modifying VBAP for more than three loudspeakers by forming an overdetermined system to determine amplitude weights for all loudspeakers at once.

Description

Technical Field

[0001] The present description relates to three-dimensional audio source spatialization in systems such as telepresence systems. Background Art

[0002] Telepresence refers to a set of technologies that allow a person to feel as if they are in person or to give the appearance of being in a place other than their actual location. For example, instead of traveling a long distance to conduct an in-person meeting, a person uses a telepresence system that uses a multi-codec video system to provide the appearance of being in an in-person meeting. Each member of the meeting "dials in" using the telepresence room and is able to see and talk to each other on the screen as if they were in the same room. Such telepresence systems can represent an improvement over conventional teleconferencing and videoconferencing because the visual aspect greatly enhances communication, allowing for the perception of facial expressions and other body language. Summary of the Invention

[0003] In one general aspect, a method can include receiving, by processing circuitry configured to perform audio source spatialization, audio data from an audio source at a source location, the audio data representing an audio waveform converted to sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location. The method can also include, in response to a frequency of the audio signal being below a specified threshold, performing, by the processing circuitry, a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of respective audio signals emitted by the loudspeakers to determine spatialization cues. The method can further include, in response to a frequency of the audio signal being above a specified threshold, performing, by the processing circuitry, a vector-based amplitude translation (VBAP) operation on the plurality of loudspeakers to generate respective weights for the loudspeakers, the respective weights for each of the plurality of loudspeakers representing a factor by which the audio signal emitted by the loudspeaker is multiplied to determine the spatialization cues. In some embodiments, the weights are complex and include a phase.

[0004] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit configured to perform audio source spatialization, causes the processing circuit to perform a method. The method can include receiving audio data from an audio source at a source location, the audio data representing an audio waveform converted to sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location. The method can also include generating a loudspeaker matrix having elements that are components of a vector parallel to a difference between the listener location and a corresponding loudspeaker location of each of the plurality of loudspeakers. The method can further include generating a source vector having elements that are components of a vector parallel to a difference between the listener location and the source location. The method can further include performing a pseudo-inverse operation on the loudspeaker matrix and the source vector to generate a weight vector having components, each component of the weight vector representing a corresponding weight for each of the plurality of loudspeakers.

[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic diagram illustrating an example electronic environment for implementing the improved techniques described herein.

[0007] Figure 2 is a flow chart illustrating an example method for performing improvement techniques within an electronic environment.

[0008] Figure 3 is a schematic diagram illustrating example geometries used in considering crosstalk cancellation (CC) operation.

[0009] Figure 4 is a schematic diagram illustrating an example rigid sphere HRTF model at two different arrival orientations.

[0010] Figure 5 is a schematic diagram illustrating example geometries used in considering a Vector Based Amplitude Translation (VBAP) operation.

[0011] Figure 6 is a flow chart illustrating an example process for performing a VBAP operation.

[0012] Figure 7 The diagram illustrates examples of computer devices and mobile computer devices that can be used with the circuits described herein. DETAILED DESCRIPTION

[0013] The goal of telepresence systems delivering this audio is to provide the listener with appropriately spatialized talker speech. Such systems accurately deliver sound to both the listener's left and right ears. This delivery is simplified if headphones are permitted. However, in interesting telepresence examples, the listening experience is unimpeded, and therefore, loudspeaker presentation is used.

[0014] There are a variety of technologies for delivering spatialized audio to listeners, including wave field synthesis and ambisonics. These technologies are typically used for rendering complex acoustic environments (with many sound sources) and require a minimum of four loudspeakers (for B-format ambisonics installations) and often more (for higher-order ambisonics and wave field synthesis installations). Furthermore, the loudspeakers used for ambisonics envelop / surround the listener.

[0015] In contrast, the telepresence systems described above use a relatively small number of loudspeakers (e.g., between two and four). In some embodiments, these loudspeakers are positioned in front of the listener. Consequently, neither multi-channel stereophonic nor wave field synthesis is suitable for use in telepresence systems described above. Instead, loudspeaker displays are centered around two conceptually simple techniques that aim to use two or more loudspeakers to display spatialized sound to a single listener: crosstalk cancellation and vector-based amplitude panning.

[0016] One conventional method of delivering audio in a telepresence system involves using crosstalk cancellation techniques to determine the complex signal from each loudspeaker that produces the desired signal at each listener's ear. Another conventional method of delivering audio in a telepresence system involves using vector-based amplitude panning (VBAP) to derive amplitude weightings for each loudspeaker that appropriately position the audio source.

[0017] The conventional methods described above for delivering audio in telepresence systems have several drawbacks that can result in poor spatialization. For example, while crosstalk cancellation can provide more accurate spatialization cues, it also tends to be sensitive to tracker errors at high frequencies, where the wavelength of the sound approaches the amplitude of the tracker error. VBAP is less sensitive to tracker errors but produces less accurate spatialization cues.

[0018] Furthermore, VBAP assumes that there are exactly three loudspeakers and that the listener's head is equidistant from each loudspeaker. If there are more than three loudspeakers, the region defined by the loudspeakers is decomposed into triangles whose vertices do not intersect the loudspeakers, and VBAP is performed on each triplet of triangles. This can be problematic because there can be more than one way to decompose the region and there's no clear way to determine which way is preferable.

[0019] According to the embodiments described herein, and in contrast to the conventional methods described above for delivering audio in a telepresence system, an improved technique for delivering audio in a telepresence system includes specifying a frequency threshold below which crosstalk cancellation (CC) is used and above which VBAP is used. In some embodiments, such a frequency threshold is between 1000 Hz and 2000 Hz. Furthermore, in some embodiments, the improved technique includes modifying VBAP for more than three loudspeakers by forming an overdetermined system to determine amplitude weights for all loudspeakers at once.

[0020] Such a hybrid scheme maintains more accurate CC localization cues in frequency regions where CC localization cues are most important and CC sensitivity to tracker errors and head-related transfer function (HRTF) individualization is lowest, while outside of these frequency regions, the less accurate and less sensitive VBAP localization cues are used. Furthermore, modified VBAP does not assume that the listener is at an equal distance from all loudspeakers, and the weights determined by modified VBAP for each loudspeaker do not depend on an arbitrary decomposition of the region spanned by those loudspeakers.

[0021] Figure 1 is a schematic diagram illustrating an example electronic environment 100 in which the above-described improved techniques may be implemented. Figure 1 In FIG. 1 , the example electronic environment 100 includes a sound rendering computer 120 .

[0022] The sound rendering computer 120 is configured to implement the above-described hybrid scheme and perform the modified VBAP operation described above. The sound rendering computer 120 includes a network interface 122, one or more processing units 124, and a memory 126. The network interface 122 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic signals and / or optical signals into electronic form for use by the sound rendering computer 120. The processing unit set 124 includes one or more processing chips and / or components. The memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid-state drives, etc. The processing unit set 124 and the memory 126 together form a control circuit that is configured and arranged to perform the various methods and functions described herein.

[0023] In some embodiments, one or more of the components of the sound rendering computer 120 can be or can include a processor (e.g., processing unit 124) configured to process instructions stored in the memory 126. Figure 1 The depicted embodiment of such instructions includes a sound acquisition manager 130, a crosstalk cancellation manager 140, and a VBAP manager 150. In addition, as Figure 1As illustrated, the memory 126 is configured to store various data, which are described with respect to the corresponding managers using such data.

[0024] The sound acquisition manager 130 is configured to acquire sound data 132 from a sound source. For example, in a telepresence system for hosting a virtual meeting, a conference participant at a remote location speaks, and the sound generated by the speech is detected by a microphone. The microphone converts the detected sound into a digital data format, which is sent to the sound rendering computer 120 via a network.

[0025] The sound data 132 represents the audio detected by the microphone and converted into a digital data format. In some embodiments, the digital data format is uncompressed, mono, has 16kHz and 16-bit resolution. In some embodiments, the digital data format uses a compressed stereo format such as Opus or MP3. In some embodiments, the recording is performed at a rate higher than 16kHz, such as 44kHz or 48kHz. In some embodiments, the resolution is higher than 16 bits, such as 24 bits, 32 bits, float, etc. The sound rendering computer 120 is then configured to convert the sound data 132 into sound played through the loudspeakers so that at the listener's position, the listener will perceive the sound as originating from a virtual source position (e.g., at the seat next to the listener).

[0026] The sound data 132 uses a waveform to represent the audio generated by a source at any moment. The waveform represents the range of frequencies at each moment or over a time window. In some embodiments, the sound acquisition manager 130 is configured to store a frequency-space representation of the sound data 132 over a specified time window (e.g., 10 seconds, 1 second, 0.5 seconds, 0.1 seconds, etc.). In this case, for each time window, there is a distribution of frequencies and corresponding amplitudes and phases.

[0027] The loudspeaker position data 134 represents the position of the loudspeaker near the listener. The position is specified relative to the origin of a specified coordinate system. In some embodiments, the origin of the coordinate system is at a point in the listener's head. In some embodiments, the loudspeaker position data is represented by a Cartesian coordinate triple.

[0028] The virtual source position data 136 represents the position of the virtual source within the aforementioned coordinate system. The position of the virtual source is the apparent location of the sound source as heard by the listener. For example, in a telepresence system, it may be desirable to conduct a conference with a remote user, but as if the remote user were sitting next to the listener. In this case, the position of the virtual source would be at that location next to the listener.

[0029] The listener position data 138 represents the listener's position within the coordinate system. In some embodiments, the listener's position is at the origin of the coordinate system. In some embodiments, the listener position data 138 changes over time, corresponding to tracking of the listener's movement.

[0030] The crosstalk cancellation manager 140 is configured to perform a crosstalk cancellation operation on the sound data 132 and the HRTF data 142 to generate amplitude / phase data 144. Figure 3 and Figure 4 As discussed in detail, the crosstalk cancellation operation generates an amplitude / phase signal at each loudspeaker based on the sound data 132 and the HRTF data 142. The operation is performed by the sound rendering computer 120 when the frequency is below a specified threshold, such as 1000 Hz, 2000 Hz, or in between.

[0031] HRTF data 142 represents various HRTFs between each loudspeaker and each ear of the listener. With two loudspeakers and two ears, there are four HRTFs for each configuration of user and loudspeaker. In some embodiments, the HRTFs are based on a rigid sphere model, i.e., a parametric model that depends on the listener's position and orientation relative to the loudspeakers. Like sound data, HRTFs are represented in frequency space.

[0032] The amplitude / phase data 144 represents the output of the crosstalk cancellation operation, i.e., the corresponding amplitude and phase of the desired sound emitted at each loudspeaker so that the listener hears the corresponding desired sound in each ear. In some embodiments, because the sound data 132 is sampled in frequency space over a time window, the amplitude / phase data 144 will change with the duration of each time window.

[0033] The VBAP manager 150 is configured to perform VBAP operations on the loudspeaker position data 134, the virtual source position data 136, and the listener position data 138 to generate weight vector data 162 representing amplitude weights for each loudspeaker. Figure 1 As shown, the VBAP manager 150 includes a loudspeaker matrix manager 152 , a source vector manager 154 , and a pseudo-inverse manager 156 .

[0034] The loudspeaker matrix manager 152 is configured to generate loudspeaker matrix data 158 based on the loudspeaker position data 134 and the listener position data 138. In some embodiments, the loudspeaker matrix data 158 has columns comprising components of unit vectors in directions of loudspeaker positions relative to listener positions.

[0035] Source vector manager 154 is configured to generate source vector data 160 based on virtual source position data 136 and listener position data 138. In some implementations, source vector data 160 has elements that include components of unit vectors in directions of virtual source positions relative to listener positions.

[0036] The pseudo-inverse manager 156 is configured to perform a pseudo-inverse operation on the loudspeaker matrix data 158 and the source vector data 160 to generate weight vector data 162. In some embodiments, the pseudo-inverse operation includes generating a Penrose pseudo-inverse from the loudspeaker matrix data 158. In some embodiments, the pseudo-inverse operation includes generating a singular value decomposition (SVD) of the loudspeaker matrix represented by the loudspeaker matrix data 158.

[0037] Weight vector data 162 represents a weight vector having elements corresponding to weights for each loudspeaker. The weight for a loudspeaker represents the factor by which the signal emitted by that loudspeaker is multiplied so that the listener hears the desired sound. In some embodiments, each element of the weight vector is a positive number. In some embodiments, at least one of the elements of the weight vector is zero, indicating that the loudspeaker corresponding to the zero weight is ineffective in producing the desired sound for the listener.

[0038] In some embodiments, the memory 126 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some embodiments, the memory 126 can be implemented as more than one memory component associated with a component of the sound rendering computer 120 (e.g., more than one RAM component or disk drive memory). In some embodiments, the memory 126 can be a database memory. In some embodiments, the memory 126 can be or include non-local memory. For example, the memory 126 can be or include memory shared by multiple devices (not shown). In some embodiments, the memory 126 can be associated with a server device (not shown) within a network and configured to serve the components of the sound rendering computer 120.

[0039] The components (e.g., modules, processing unit 124) of the sound rendering computer 120 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some embodiments, the components of the sound rendering computer 120 can be configured to operate within a cluster of devices (e.g., a server farm). In such an implementation, the functionality and processing of the components of the sound rendering computer 120 can be distributed across several devices of the cluster of devices.

[0040] The components of the sound rendering computer 120 can be or can include any type of hardware and / or software configured to process attributes. Figure 1 One or more portions of the components shown in the components of the sound rendering computer 120 in FIG. 1 can be or can include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions that can be executed at a computer). For example, in some embodiments, one or more portions of the components of the sound rendering computer 120 can be or can include software modules configured for execution by at least one processor (not shown). In some embodiments, the functionality of the components can be included in conjunction with Figure 1 those shown in different modules and / or different components.

[0041] Although not shown, in some embodiments, the components of the sound rendering computer 120 (or portions thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some embodiments, the components of the sound rendering computer 120 (or portions thereof) can be configured to operate within a network. Thus, the components of the sound rendering computer 120 (or portions thereof) can be configured to function in various types of network environments that can include one or more devices and / or one or more server devices. For example, the network can be or can include a local area network (LAN), a wide area network (WAN), etc. The network can be or can include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network can include one or more segments and / or can have portions based on various protocols such as the Internet Protocol (IP) and / or proprietary protocols. The network can include at least a portion of the Internet.

[0042] In some embodiments, one or more of the components of the sound rendering computer 120 can be or include a processor configured to process instructions stored in a memory. For example, the sound acquisition manager 130 (and / or a portion thereof), the crosstalk cancellation manager 140 (and / or a portion thereof), and the VBAP manager 150 (and / or a portion thereof) can be a combination of a processor and memory configured to execute instructions associated with a process for implementing one or more functions.

[0043] Figure 2 is a flow chart illustrating an example method 200 for mapping user interaction data into discrete buckets. The method 200 may be performed in conjunction with Figure 1The software constructs described together are executed, resident in the memory 126 of the sound rendering computer 120 and run by the set of processing units 124 .

[0044] At 202 , the sound acquisition manager 130 receives audio data from an audio source at a source location, the audio data representing an audio waveform converted into sound of certain frequencies via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location.

[0045] At 204 , the crosstalk cancellation manager 140 performs a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of respective audio signals emitted by the loudspeakers to determine spatialization cues in response to the frequency of the audio signal being below a specified threshold.

[0046] At 206 , the VBAP manager 150 performs VBAP operations on the plurality of loudspeakers to generate respective weights for the loudspeakers in response to the frequency of the audio signal being above a specified threshold, the respective weight for each of the plurality of loudspeakers representing a factor by which the audio signal emitted by the loudspeaker is multiplied to determine the spatialization cues.

[0047] Figure 3 is a schematic diagram illustrating an example geometry 300 for use when considering crosstalk cancellation (CC) operation. Within the geometry 300, a pair of loudspeakers 310(1) and 310(2) face a listener 320.

[0048] The propagation of sound from a source to a human listener is often described in terms of a head-related transfer function (HRTF). An HRTF is a frequency response that describes the propagation from a point source at a specific location to the left and right ears in the absence of reverberation. An HRTF depends on many factors. For simplicity, it is often simplified to depend on the source arrival orientation relative to the direction the head is pointing—i.e., azimuth and elevation. Other factors, such as distance and rotation of the head relative to the torso, are often ignored.

[0049] The sound presented by the loudspeaker 310(1) is transmitted using (H 1L ,H 1R ) is propagated to the two ears of the listener 320. Similarly, the sound presented by the loudspeaker 310(2) is presented as a 2L ,H 2R ) to the two ears of the listener 320 as described. This means that - expressed in the frequency domain - the signals S1 and S2 played from the loudspeakers produce observed signals L and R that obey the following relationship:

[0050]

[0051] Assume that the desired binaural signal to be presented at both ears is represented by L des and R des Given, then this system of equations can be solved to obtain the appropriate S1 and S2 that, when played through the loudspeakers, will produce the desired signal at each ear:

[0052]

[0053] Therefore, if the HRTF (H 1L ,H 1R ) and (H 2L ,H 2R ) is known, one can generate the loudspeaker output signals necessary to deliver the spatialized audio to the listener 320.

[0054] Note that when the listener's position changes relative to the loudspeakers (or vice versa), the HRTF will change. Figure 4 An example of an HRTF that can change in real time as the listener moves is provided in .

[0055] Figure 4 The HRTFs for two source orientations (az, el): (-10°, 0°) and (20°, 0°) located to the left and right of the listener's head, respectively, are shown. The top row of panels shows the amplitudes of the left and right ear transfer functions. The middle row of panels shows the amplitude of the left ear divided by the right ear frequency response. The bottom row shows the time delay relative to the left ear to the right ear. These graphs show the following HRTF characteristics that are relevant for sound localization.

[0056] The interaural time difference (ITD) is the apparent relative delay in the source signal between the two ears. Figure 4 Bottom row of panels in . A source arriving from the listener's left - i.e. from (-10°, 0°) - arrives first at the left ear and second at the right ear. This produces the negative relative delay L / R (=ITD) observed for this source position. A source arriving from the listener's right - i.e. from (20°, 0°) - shows the opposite behavior. The |ITD| of the more laterally positioned source from (20°, 0°) is larger than the |ITD| of the source from (-10°, 0°). The ITD is not constant with frequency, as it is for a point in the free field. The presence of the head produces larger ITD amplitudes at lower frequencies than at higher frequencies.

[0057] The interaural level difference (ILD) is the relative level difference in the source signal between the two ears. Figure 4, 0°). A source arriving from the listener's left side—i.e., from (-10°, 0°)—is louder at the left ear than at the right ear because the head "shadows" the source as it travels from the source to the right ear. For this source position, this produces a directly proportional value of the amplitude L / R (=ILD) expressed in dB. A source arriving from the listener's right side—i.e., from (20°, 0°)—shows the opposite behavior. The |ILD| of a more laterally positioned source from (20°, 0°) is typically greater than the |ILD| of a source from (-10°, 0°) because the degree of head shadowing is higher. Similar to ITD, ILD is not constant with frequency. The presence of the head produces an ILD amplitude that is larger at higher frequencies than at lower frequencies.

[0058] As in Figure 4 The spectral clues to the distinct peaks, valleys, and notches in the transfer function amplitude are shown in the top row of panels in [ 1 ]. These arise from a variety of factors including ear canal resonance, reflections from the listener's torso / shoulders, and reflections from the outer ear or pinna.

[0059] Typically, interaural cues (ITD and ILD) reflect source lateralization (i.e., a shift to the left or right side of the listener). The broad trends in ITD and ILD are similar across different listeners and even make them easy to simulate using a rigid spherical head model. Since ITD begins to alias at higher frequencies, ITD is most relevant at lower frequencies (below ~1500 Hz). Mainly due to the reduced relevance of ITD at these frequencies, ILD is most relevant at higher frequencies (above ~1500 Hz).

[0060] Interaural cues become ambiguous when considered along a "cone of confusion" of similarly lateralized source positions. For example, sources located at (az,el) = (45°, 0°), (135°, 0°), (90°, 45°), and (90°, -45°) are all similarly lateralized along the cone formed by rotating a ray pointing to (45°, 0°) about the interaural axis. Spectral cues are typically used by listeners to distinguish source positions along the same cone of confusion. In particular, spectral cues are useful for elevation localization and front / rear source discrimination. They are also useful for "externalization" - that is, making a sound appear as if it originates from an actual point outside the head. Due to the highly individual variations in the structure of the pinna across different listeners, spectral cues are highly individualized.

[0061] The telepresence system is configured to render the remote talker's speech as if the talker were in the listener's acoustic space. It is assumed that the sound rendering computer 120 has appropriately "cleaned up" the transmitted audio so that it is a single channel consisting only of the talker's speech. The task of the sound rendering computer 120 is to convert this single source into a binaural signal based on the relative positions and head orientations of the listener and talker. This is done by Figure 3 This is accomplished by applying the appropriate HRTF to the talker's speech as shown to produce the signal that should be presented to the listener's ears.

[0062] One technique for obtaining these signals is the rigid sphere model or rigid sphere HRTF model for ILD / ITD rendering. Studies have shown that the rigid sphere model is able to produce interaural cues, particularly ITD, that mirror those observed with actual listeners. Figure 4 The synthetic HRTF based on a rigid sphere head model with a radius of 8.5 cm is also shown in dashed lines. (Other radii can be used, such as 8.0 cm, 9.0 cm, 7.5 cm, 9.5 cm, etc.) The interaural cues are very similar, but the high-frequency ILD tends to be reduced. There are no detailed spectral cues, but this is not surprising. However, the rigid sphere model has the advantage of being fully parameterized and mathematically solvable.

[0063] Another technique that can be used is custom HRTF rendering, in which the listener's own empirically derived HRTFs are applied. While this produces the most accurate and realistic binaural signals, in some embodiments, the costs associated with this approach render it impractical as a general approach.

[0064] Another technique that can be used is reference set HRTF rendering. Instead of using the HRTF of an individual listener, an alternative would be to use a generic "typical" HRTF for spatialization, or an HRTF selected from a library of reference HRTFs. Since the interaural cues for ITD and ILD are generally similar across listeners, this will result in good spatialization, especially with respect to lateral sources.

[0065] Another technique that can be used is reference set ILD / ITD rendering. Instead of using full HRTFs to synthesize spatialization, a simpler alternative would be to synthesize only the interaural (ITD and ILD) localization cues. These cues are similar across listeners, so the use of interaural cues or a "reference set" will produce a similar spatialization of lateral sources as achieved using the listener's own interaural cues. Furthermore, interaural cues are generally less "rich" than full HRTFs, which means they can be parameterized or sampled in a less dense set of source orientations, thereby reducing memory usage at runtime.

[0066] As mentioned above, the CC operation described above is preferably performed for lower frequencies (e.g., below between 1000 Hz and 2000 Hz). Above such frequencies, the improved technique includes performing a modified VBAP operation to generate a positive set of weights for at least some of the loudspeakers.

[0067] Figure 5is a schematic diagram illustrating an example geometry 500 used when considering a modified vector-based amplitude panning (VBAP) operation. In the geometry 500, there are four loudspeakers 510(1), 510(2), 510(3), and 510(4) aimed at a listener 530. There is also typically a virtual source 520 in front of the listener 530. The listener 530 is not necessarily equidistant from all of the loudspeakers 510(1-4) and can move around relative to them. In some embodiments, there are more than four loudspeakers near the listener 530. In some embodiments, there are two loudspeakers near the listener 530.

[0068] Figure 5 Shown are loudspeakers 510 (1-4) directed from the center of the listener 530 (or generally, the listener 530). HL，1-4 and Virtual Source 520U HV From these unit vectors, the VBAP manager 150 generates an overdetermined (or underdetermined when the number of loudspeakers is less than three) linear system that produces weights corresponding to each of the loudspeakers 510 (1-4).

[0069] The linear system solution used for conventional VBAP has several limitations. First, conventional VBAP assumes that the head of the listener 530 is positioned equidistant from all loudspeakers, such as 510(1-4). Second, conventional VBAP uses exactly three loudspeakers to spatialize the virtual source 520. When there are more than three loudspeakers, conventional VBAP requires dividing the listener space into non-overlapping triangles so that each sub-area is covered by exactly three loudspeakers. In conventional VBAP, although spatialization is achieved by calculating VBAP weights for the appropriate subset of loudspeakers, it requires an arbitrary division of space into triangles. For example, when the loudspeakers 510(1-4) are arranged in a square centered on the listener, the square can be divided into two triangles in two different ways: 510(1,2,3)+510(2,3,4) or 510(1,2,4)+510(1,3,4); it is unclear which is more preferable. Furthermore, the division into groups of three loudspeakers can lead to counterintuitive loudspeaker weightings. For example, consider the square geometry above divided into two triangular sub-regions spanned by 510(1,2,3)+510(2,3,4). In this case, a virtual source located exactly in the center of the square will only have non-zero VBAP weights for loudspeakers 510(2) and 510(3). A more intuitive VBAP weighting would have equal contributions from all four loudspeakers. Third, there is no guarantee that all weights found according to conventional VBAP will be all positive. Thus, the modified VBAP with respect to Figure 4 is presented.

[0070] Figure 6 is a flow chart illustrating an example method 600 for performing modified VBAP. The method 600 may be performed by Figure 1 The described software constructs are executed by software constructs that reside in the memory 126 of the sound rendering computer 120 and are executed by the set of processing units 124.

[0071] At 602, the loudspeaker manager 152 generates a signal based on a unit vector U HL，1-4 To generate the loudspeaker matrix. Typically, the loudspeaker matrix has a unit vector in three dimensions corresponding to each loudspeaker for each column. For example, when there are N loudspeakers, the loudspeaker matrix has dimensions 3×N. For Figure 5 In the illustrated case, the matrix has dimensions 3 × 4. Therefore, the linear system is overdetermined.

[0072] At 604, the source vector manager 154 generates a source vector. In this case the source vector is simply the unit vector U HV .

[0073] At 606, the pseudo-inverse manager 156 performs a pseudo-inverse operation on the loudspeaker matrix and the source vector to generate a weight vector. For example, in some embodiments, the pseudo-inverse manager 156 calculates the matrix (L T L) -1 L T to generate the Penrose pseudo-inverse of the loudspeaker matrix L. In this case, the weights are then calculated from the quantity (L T L) -1 L T U HV The weight vector of the overdetermined system is not uniquely determined. In this case, the pseudo-inverse manager 156 generates a weight vector w with the minimum norm, that is, the sum of the squares of the components of the weight vector w is minimized.

[0074] At 608, the VBAP manager 150 determines whether all components of the weight vector are positive. If all weights are positive, the method 600 completes at 614. If not, the VBAP manager sets all components of the weight vector w to zero at 610. In effect, the VBAP manager 150 removes those loudspeakers corresponding to negative weights. In this case, at 612, the loudspeaker matrix manager 152 generates a new loudspeaker matrix L′ with columns corresponding to the removed negative weights. The method 600 then repeats until all components of the weight vector w are positive.

[0075] In some embodiments, after generating a weight vector with all positive components, the VBAP manager 150 multiplies each component by the corresponding head-to-loudspeaker distance. This multiplication corrects for the inverse-square distance speaker energy loss caused by waves propagating over different distances. In the absence of reverberation, this compensates for the non-reverberant path signal pointed at the loudspeaker for situations where the listener is at unequal distances from the loudspeaker. In some embodiments, the weight vector w may also include a phase component based on the distance between the listener and the loudspeaker. In this case, such a phase component aligns the phase of the signal at the listener's head.

[0076] The modified VBAP described above addresses all three of the above concerns. Specifically, (i) the modified VBAP does not assume that the listener is equidistant from all loudspeakers, (ii) the modified VBAP is applicable to 2+ loudspeakers, (iii) the subset of loudspeakers is selected through an iterative process rather than by arbitrarily pre-dividing the space into triangles, (iv) for an arrangement such as a square, a source located at the center of the square receives equal VBAP contributions from all four vertex loudspeakers, and (v) all weights are positive.

[0077] The improved technique described above uses the tracked listener head position to continuously update the VBAP weights for correct source spatialization. Note that VBAP depends only on the listener head position and the virtual source positions. VBAP does not require knowledge of head rotation or HRTFs. This can result in less accurate spatialization cues than those provided by CC, but the spatialization cues are also less susceptible to tracking errors and HRTF inaccuracies.

[0078] In summary, CC requires knowledge of both the listener position / rotation and the listener HRTF. VBAP, on the other hand, only requires knowledge of the listener position. In general, CC provides more accurate localization cues, but is more sensitive to tracker (especially rotation) errors and is limited by the accuracy of the underlying HRTF model, while VBAP provides less accurate localization cues but is less sensitive to tracker errors and does not require HRTF knowledge at all. CC's sensitivity to tracker errors is wavelength dependent - as the wavelength decreases, the tracker error becomes a larger fraction of the wavelength. Furthermore, the highly individual aspects of the listener HRTF are concentrated in high-frequency spectral cues that depend on the shape of the outer ear (or pinna) of the individual listener. Finally, sound localization (especially left / right localization) is dominated by low-frequency interaural cues.

[0079] These properties suggest a hybrid CC / VBAP approach that uses CC in the low-frequency region and VBAP in the high-frequency region. That way, the more accurate CC localization cues are maintained in the frequency region where they are most important and CC is least sensitive to tracker errors and HRTF individualization, while VBAP localization cues, which are less sensitive to tracker errors, are used elsewhere. A typical cutoff frequency between the low-frequency region and the high-frequency region is in the range of 1000-2000 Hz (this reflects the fact that interaural time differences begin to spatially alias in this region).

[0080] Figure 7 Illustrated are examples of a general-purpose computer device 700 and a general-purpose mobile computing device 750 that may be used with the techniques described herein.

[0081] like Figure 7 As shown in FIG, computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit embodiments of the inventions described and / or claimed in this document.

[0082] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 702 is capable of processing instructions for execution within the computing device 700, including instructions for displaying graphical information for a GUI on an external input / output device such as a display 716 coupled to the high-speed interface 708, stored in the memory 704 or on the storage device 706. In other embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memory can be used as appropriate. In addition, multiple computing devices 700 can be connected, with each device providing part of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).

[0083] Memory 704 stores information within computing device 700. In one embodiment, memory 704 is one or more volatile memory units. In another embodiment, memory 704 is one or more non-volatile memory units. Memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0084] The storage device 706 can provide mass storage for the computing device 700. In one implementation, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0085] The high-speed controller manages bandwidth-intensive operations of the computing device 700, while the low-speed controller manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 710 that can accept various expansion cards (not shown). In this embodiment, the low-speed controller is coupled to the storage device 706 and the low-speed expansion port. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0086] As shown in the figure, computing device 700 can be implemented in many different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown) (such as computing device 750). Each of such devices can contain one or more of computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0087] Computing device 750 includes a processor 752, a memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768, among other components. Computing device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 is interconnected using various buses, and several components may be mounted on a common motherboard or otherwise, as appropriate.

[0088] Processor 752 is capable of executing instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset of chips that include separate or multiple analog and digital processors. For example, the processor may provide functions for coordinating other components of computing device 750, such as controlling a user interface, applications executed by computing device 750, and wireless communications performed by computing device 750.

[0089] The processor 752 can communicate with the user through a control interface 758 and a display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them for submission to the processor 752. In addition, an external interface 762 in communication with the processor 752 can be provided to enable the computing device 750 to communicate with near-area devices of other devices. The external interface 762 can, for example, be provided for wired communication in some embodiments, or for wireless communication in other implementations, and multiple interfaces can also be used.

[0090] Memory 764 stores information within computing device 750. Memory 764 can be implemented as one or more of a computer-readable medium or media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 774 can also be provided and connected to computing device 750 via expansion interface 772, which can include, for example, a SIMM (Single In-line Memory Module) card interface. This expansion memory 774 can provide additional storage space for computing device 750 or store applications or other information for computing device 750. Specifically, expansion memory 774 can include instructions for executing or supplementing the processes described above, and can also include security information. Thus, for example, expansion memory 774 can be provided as a security module for computing device 750 and can be programmed with instructions that allow for secure use of computing device 750. Furthermore, security applications, along with additional information, can be provided via a SIMM card, such as placing identification information on the SIMM card in an unhackable manner.

[0091] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which can be received, for example, via transceiver 768 or external interface 762.

[0092] The computing device 750 can communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry, if necessary. The communication interface 766 may provide for communicating in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communication may occur, for example, via a radio frequency transceiver 768. In addition, short-range communication may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 may provide additional navigation and location-related wireless data to the computing device 750, which may be used as appropriate by applications running on the computing device 750.

[0093] Computing device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through a speaker in an earpiece of computing device 750, for example. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on computing device 750.

[0094] As shown in the figure, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile device.

[0095] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor that can be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0096] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium," "computer-readable medium," and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0097] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0098] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser that a user can use to interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0099] A computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0100] A number of embodiments have been described, however, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure.

[0101] It should also be understood that when an element is referred to as being on another element, being connected to, being electrically connected to, being coupled to or being electrically coupled to another element, it can be directly on another element, being connected or being coupled to another element, or one or more intermediate elements can be present.On the contrary, when an element is referred to as being directly on another element, being directly connected to or being directly coupled to another element, there is no intermediate element.Although the term "directly on", "being directly connected to" or "being directly coupled to" can not be used in the entire detailed description, the element that is shown as being directly on, being directly connected or being directly coupled can be referred to as such.The claims of the present application can be modified to narrate the exemplary relationship described in the specification or shown in the drawings.

[0102] Although certain features of the described embodiments have been shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It should be understood that they have been presented only as examples and not by way of limitation, and that various changes in form and detail may be made. Any part of the apparatus and / or method described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein can include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.

[0103] Additionally, the logic flows depicted in the accompanying drawings do not require the particular order shown or sequential order to achieve the desired results. Furthermore, other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method for spatializing an audio source, comprising: receiving, by processing circuitry configured to perform audio source spatialization, audio data from an audio source at a source location, the audio data representing an audio waveform converted to sound of frequencies via a plurality of loudspeakers configured to be heard by a listener at a listener location; In response to the frequency of the audio signal being below a specified threshold, performing, by the processing circuit, a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of corresponding audio signals emitted by the plurality of loudspeakers to determine spatialization cues; as well as In response to the frequency of the audio signal being higher than the specified threshold, the processing circuit performs a vector-based amplitude panning (VBAP) operation on the plurality of loudspeakers to generate corresponding weights for loudspeakers among the plurality of loudspeakers, the weights for the loudspeakers representing factors, the audio signal emitted by the loudspeakers being multiplied by the factors to determine spatialization cues, and if the weights generated by the VBAP operation are less than zero, the VBAP operation sets the corresponding weights for the loudspeakers to zero.

2. The method according to claim 1, wherein: Performing the CC operations on the plurality of loudspeakers includes tracking the position and orientation of the listener over time.

3. The method according to claim 1, wherein: The number of loudspeakers in the plurality of loudspeakers is an even number, and The performing the CC operation on the plurality of loudspeakers comprises: applying a head related transfer function HRTF configured to provide a binaural sound field to the listener to the corresponding loudspeakers, wherein the HRTF is based on a parameterized rigid sphere model.

4. The method according to any one of claims 1 to 3, wherein: The specified threshold is between 1000 Hz and 2000 Hz.

5. The method according to claim 1, wherein: Performing the VBAP operation on the plurality of loudspeakers includes: generating a loudspeaker matrix having elements that are components of a vector parallel to a difference between the listener position and the loudspeaker position; generating a source vector having elements that are components of a vector parallel to a difference between the listener position and the source position; and A pseudo-inverse operation is performed on the loudspeaker matrix and the source vector to generate a weight vector.

6. The method according to claim 5, wherein: A distance between the listener and a first loudspeaker of the plurality of loudspeakers is different from a distance between the listener and a second loudspeaker of the plurality of loudspeakers.

7. The method according to claim 5, wherein: The number of loudspeakers in the plurality of loudspeakers is greater than three, and The performing of the pseudo-inverse operation on the microphone matrix and the source vector comprises generating a product of the microphone inverse and the Penrose pseudo-inverse of the source vector.

8. The method according to claim 7, wherein: Performing the pseudo-inverse operation on the loudspeaker matrix and the source vector further includes minimizing the sum of squares of components of the weight vector.

9. The method according to claim 7, wherein: The components of the weight vector are less than zero, and Wherein, the method further comprises: removing elements of the loudspeaker matrix corresponding to the loudspeakers corresponding to the components of the weight vector that are less than zero to form a reduced loudspeaker matrix; and The pseudo-inverse operation is performed on the reduced loudspeaker matrix and the source vector to produce a reduced weight vector.

10. The method according to any one of claims 5 to 9, further comprising: Each of the components of the weight vector is multiplied by a corresponding scaling factor, the scaling factor being proportional to a distance between the listener and the loudspeaker of the plurality of loudspeakers to which the component of the weight vector corresponds.

11. An electronic device configured to perform audio source spatialization, the electronic device comprising: Memory; as well as a control circuit coupled to the memory, the control circuit being configured to: receiving audio data from an audio source at a source location, the audio data representing an audio waveform that is converted to sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location; In response to the frequency of the audio signal being below a specified threshold, performing a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of respective audio signals emitted by the plurality of loudspeakers to determine spatialization cues; as well as In response to the frequency of the audio signal being above the specified threshold, performing a vector-based amplitude panning (VBAP) operation on the plurality of loudspeakers to generate corresponding weights for loudspeakers among the plurality of loudspeakers, the weights for the loudspeakers representing factors, the audio signal emitted by the loudspeakers being multiplied by the factors to determine spatialization cues, and the VBAP operation setting the corresponding weights for the loudspeakers to zero if the weights generated by the VBAP operation are less than zero.

Images