Three-dimensional audio source spatialization

Abstract

The invention relates to three-dimensional audio source spatialization. Techniques for delivering audio in a telepresence system include specifying a frequency threshold below which crosstalk cancellation (CC) is used and above which VBAP is used. In some embodiments, such a frequency threshold is between 1000 Hz and 2000 Hz. Further, in some embodiments, the improved techniques include modifying a VBAP for more than three loudspeakers by forming an overdetermined system to determine amplitude weights for all loudspeakers at a time.

Description

Description of the case This application is a divisional application of Chinese invention patent application No. 201980095340.8, with a filing date of June 12, 2019. Technical Field

[0001] The present description relates to three-dimensional audio source spatialization in a system such as a telepresence system. Background Art

[0002] Telepresence refers to a set of technologies that allow a person to feel as if they are in person or to give the appearance of being in a place other than their actual location. For example, instead of traveling a long distance to conduct a face-to-face meeting, a person uses a telepresence system that uses a multi-codec video system to provide the appearance of being in a face-to-face meeting. Each member of the meeting "dials in" using the telepresence room and is able to see and talk to each other member on the screen as if they were in the same room. Such telepresence systems can represent an improvement over conventional teleconferencing and videoconferencing because the visual aspect greatly enhances the communication, allowing for the perception of facial expressions and other body language. Summary of the invention

[0003] In one general aspect, a method can include receiving, by a processing circuit configured to perform audio source spatialization, audio data from an audio source at a source location, the audio data representing an audio waveform that is converted to a sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location. The method can also include, in response to the frequency of the audio signal being below a specified threshold, performing, by the processing circuit, a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate an amplitude and phase of a corresponding audio signal emitted by the loudspeaker to determine a spatialization cue. The method can further include, in response to the frequency of the audio signal being above a specified threshold, performing, by the processing circuit, a vector-based amplitude translation (VBAP) operation on the plurality of loudspeakers to generate a corresponding weight for the loudspeaker, the corresponding weight for each of the plurality of loudspeakers representing a factor by which the audio signal emitted by the loudspeaker is multiplied to determine the spatialization cue. In some implementations, the weight is complex and includes a phase.

[0004] In another general aspect, a computer program product including a non-transitory storage medium includes code that, when executed by a processing circuit configured to perform audio source spatialization, causes the processing circuit to perform a method. The method can include receiving audio data from an audio source at a source location, the audio data representing an audio waveform that is converted to a sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location. The method can also include generating a loudspeaker matrix having elements that are components of a vector parallel to a difference between the listener location and the corresponding loudspeaker location of each of the plurality of loudspeakers. The method can further include generating a source vector having elements that are components of a vector parallel to a difference between the listener location and the source location. The method can further include performing a pseudo-inverse operation on the loudspeaker matrix and the source vector to generate a weight vector having components, each component of the weight vector representing a corresponding weight for each of the plurality of loudspeakers.

[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic diagram illustrating an example electronic environment for implementing the improved techniques described herein.

[0007] Figure 2 is a flow chart illustrating an example method of performing improvement techniques within an electronic environment.

[0008] Figure 3 is a schematic diagram illustrating example geometries used in considering crosstalk cancellation (CC) operations.

[0009] Figure 4 is a schematic diagram illustrating an example rigid sphere HRTF model at two different arrival orientations.

[0010] Figure 5 is a schematic diagram illustrating example geometries used in considering a vector-based amplitude translation (VBAP) operation.

[0011] Figure 6 is a flow chart illustrating an example process for performing a VBAP operation.

[0012] Figure 7 Examples of computer devices and mobile computer devices that can be used with the circuits described herein are shown. DETAILED DESCRIPTION

[0013] The goal of a telepresence system that delivers the above audio is to provide the listener with a properly spatialized talker voice. Such a system accurately delivers sound to the left and right ears of the listener. If the use of headphones is permitted, the delivery would be simple. However, in the telepresence example of interest, the listening experience is not hindered, and therefore, loudspeaker presentation is used.

[0014] There are a variety of techniques for delivering spatialized audio to a listener - including wave field synthesis and stereophony. These techniques are typically used for rendering of complex acoustic environments (with many sound sources) and require a minimum of four (for B-format stereophonic installations) and often more (for higher-order stereophonic and wave field synthesis installations) loudspeakers. Furthermore, the loudspeakers for stereophonic envelop / surround the listener.

[0015] In contrast, the telepresence systems described above use a relatively small number of loudspeakers (e.g., between two and four). In some embodiments, these loudspeakers are positioned in front of the listener. Therefore, neither multi-channel stereophonic nor wave field synthesis is suitable for use in the telepresence systems described above. Instead, loudspeaker displays are centered around two conceptually simple techniques that intend to use two or more loudspeakers to display spatialized sound to a single listener: crosstalk cancellation and vector-based amplitude panning.

[0016] One conventional method of delivering audio in a telepresence system includes using crosstalk cancellation techniques to determine the complex signal from each loudspeaker that produces the desired signal in each listener's ear. Another conventional method of delivering audio in a telepresence system includes using vector-based amplitude panning (VBAP) to derive amplitude weightings for each loudspeaker that appropriately position the audio source.

[0017] The above conventional methods of delivering audio in telepresence systems have some drawbacks that may result in poor spatialization. For example, while crosstalk cancellation can provide more accurate spatialization cues, crosstalk cancellation also tends to be sensitive to tracker errors at high frequencies, where the sound wavelength is close to the amplitude of the tracker error. VBAP is less sensitive to tracker errors, but produces less accurate spatialization cues.

[0018] Furthermore, VBAP assumes that there are exactly three loudspeakers and that the listener's head is equidistant from each loudspeaker. If there are more than three loudspeakers, the region defined by the loudspeakers is decomposed into triangles that do not intersect the loudspeakers at their vertices, and VBAP is performed for each triplet of triangles. This can be problematic because there can be more than one way to decompose the region and there is no clear way to determine which way is preferable.

[0019] In accordance with the embodiments described herein and in contrast to the above-described conventional methods of delivering audio in a telepresence system, an improved technique for delivering audio in a telepresence system includes specifying a frequency threshold below which crosstalk cancellation (CC) is used and above which VBAP is used. In some embodiments, such a frequency threshold is between 1000 Hz and 2000 Hz. Furthermore, in some embodiments, the improved technique includes modifying VBAP for more than three loudspeakers by forming an overdetermined system to determine amplitude weights for all loudspeakers at once.

[0020] Such a hybrid scheme maintains more accurate CC localization cues in frequency regions where CC localization cues are most important and CC sensitivity to tracker errors and head-related transfer function (HRTF) individualization is lowest, while outside of frequency regions the less accurate and less sensitive VBAP localization cues are used. Furthermore, the modified VBAP does not assume that the listener is at an equal distance from all loudspeakers, and the weights determined by the modified VBAP for each loudspeaker do not depend on an arbitrary decomposition of the area spanned by those loudspeakers.

[0021] Figure 1 1 is a schematic diagram illustrating an example electronic environment 100 in which the above-described improved techniques may be implemented. Figure 1 , the example electronic environment 100 includes a sound rendering computer 120 .

[0022] The sound rendering computer 120 is configured to implement the above-mentioned hybrid scheme and perform the above-mentioned modified VBAP operation. The sound rendering computer 120 includes a network interface 122, one or more processing units 124 and a memory 126. The network interface 122 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic signals and / or optical signals into electronic form for use by the sound rendering computer 120. The processing unit set 124 includes one or more processing chips and / or components. The memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid-state drives, etc. The processing unit set 124 and the memory 126 together form a control circuit that is configured and arranged to perform various methods and functions as described herein.

[0023] In some embodiments, one or more of the components of the sound rendering computer 120 can be or can include a processor (e.g., processing unit 124) configured to process instructions stored in the memory 126. Figure 1 The depicted embodiment of such instructions includes a sound acquisition manager 130, a crosstalk cancellation manager 140, and a VBAP manager 150. In addition, as Figure 1As illustrated, the memory 126 is configured to store various data, which are described with respect to the corresponding managers using such data.

[0024] The sound acquisition manager 130 is configured to acquire sound data 132 from a sound source. For example, in a telepresence system for hosting a virtual meeting, a conference participant at a remote location speaks, and the sound generated by the speech is detected by a microphone. The microphone converts the detected sound into a digital data format, which is sent to the sound rendering computer 120 via a network.

[0025] The sound data 132 represents the audio detected by the microphone and converted into a digital data format. In some embodiments, the digital data format is uncompressed, mono, with 16 kHz and 16-bit resolution. In some embodiments, the digital data format adopts a compressed stereo format such as Opus or MP3. In some embodiments, the recording is performed at a rate higher than 16 kHz, such as 44 kHz or 48 kHz. In some embodiments, the resolution is higher than 16 bits, such as 24 bits, 32 bits, floating, etc. The sound rendering computer 120 is then configured to convert the sound data 132 into sound played through the loudspeaker so that at the listener's position, the listener will perceive the sound as originating from a virtual source position (e.g., at the seat next to the listener).

[0026] The sound data 132 uses a waveform to represent the audio produced by the source at any moment. The waveform represents the range of frequencies at each moment or throughout a time window. In some embodiments, the sound acquisition manager 130 is configured to store a frequency space representation of the sound data 132 throughout a specified time window (e.g., 10 seconds, 1 second, 0.5 seconds, 0.1 seconds, etc.). In this case, for each time window, there is a distribution of frequencies and corresponding amplitudes and phases.

[0027] The microphone position data 134 indicates the position of the microphone near the listener. The position is specified with respect to the origin of a specified coordinate system. In some embodiments, the origin of the coordinate system is at a point in the listener's head. In some embodiments, the microphone position data is represented by a Cartesian coordinate triple.

[0028] The virtual source position data 136 represents the position of the virtual source within the above-mentioned coordinate system. The position of the virtual source is the apparent position of the sound source as heard by the listener. For example, in a telepresence system, it may be desirable to conduct a conference with a remote user, but as if the remote user were sitting next to the listener. In this case, the position of the virtual source would be at that place next to the listener.

[0029] The listener position data 138 represents the position of the listener within the above-mentioned coordinate system. In some embodiments, the listener's position is at the origin of the coordinate system. In some embodiments, the listener position data 138 changes over time, corresponding to the tracking of the listener's movement.

[0030] The crosstalk cancellation manager 140 is configured to perform a crosstalk cancellation operation on the sound data 132 and the HRTF data 142 to generate amplitude / phase data 144. Figure 3 and Figure 4 As discussed in detail, the crosstalk cancellation operation generates an amplitude / phase signal at each loudspeaker based on the sound data 132 and the HRTF data 142. The operation is performed by the sound rendering computer 120 when the frequency is below a specified threshold, such as 1000 Hz, 2000 Hz, or in between.

[0031] The HRTF data 142 represents various HRTFs between each speaker and each ear of the listener. With two loudspeakers and two ears, there are four HRTFs for each configuration of user and loudspeaker. In some embodiments, the HRTFs are based on a rigid sphere model, i.e., a parametric model that depends on the position and orientation of the listener relative to the loudspeakers. Like sound data, the HRTFs are represented in frequency space.

[0032] The amplitude / phase data 144 represents the output of the crosstalk cancellation operation, i.e., the corresponding amplitude and phase emitted at each loudspeaker so that the listener hears the corresponding desired sound in each ear. In some embodiments, because the sound data 132 is sampled in frequency space over a time window, the amplitude / phase data 144 will change with the duration of each time window.

[0033] The VBAP manager 150 is configured to perform VBAP operations on the loudspeaker position data 134, the virtual source position data 136, and the listener position data 138 to generate weight vector data 162 representing amplitude weights for each loudspeaker. Figure 1 As shown, the VBAP manager 150 includes a loudspeaker matrix manager 152 , a source vector manager 154 , and a pseudo-inverse manager 156 .

[0034] The loudspeaker matrix manager 152 is configured to generate loudspeaker matrix data 158 based on the loudspeaker position data 134 and the listener position data 138. In some embodiments, the loudspeaker matrix data 158 has columns including components of unit vectors in directions of loudspeaker positions relative to listener positions.

[0035] The source vector manager 154 is configured to generate source vector data 160 based on the virtual source position data 136 and the listener position data 138. In some implementations, the source vector data 160 has elements that include components of unit vectors in directions of the virtual source positions relative to the listener positions.

[0036] The pseudo-inverse manager 156 is configured to perform a pseudo-inverse operation on the loudspeaker matrix data 158 and the source vector data 160 to generate the weight vector data 162. In some embodiments, the pseudo-inverse operation includes generating a Penrose pseudo-inverse from the loudspeaker matrix data 158. In some embodiments, the pseudo-inverse operation includes generating a singular value decomposition (SVD) of the loudspeaker matrix represented by the loudspeaker matrix data 158.

[0037] The weight vector data 162 represents a weight vector having elements that are corresponding weights for each loudspeaker. The weight for a loudspeaker represents the factor by which the signal emitted by the loudspeaker is multiplied so that the listener hears the desired sound. In some embodiments, each element of the weight vector is a positive number. In some embodiments, at least one of the elements of the weight vector is zero, suggesting that the loudspeaker corresponding to the zero weight does not work in producing the desired sound for the listener.

[0038] In some embodiments, the memory 126 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some embodiments, the memory 126 can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with a component of the sound rendering computer 120. In some embodiments, the memory 126 can be a database memory. In some embodiments, the memory 126 can be or can include a non-local memory. For example, the memory 126 can be or can include a memory shared by multiple devices (not shown). In some embodiments, the memory 126 can be associated with a server device (not shown) within a network and configured to serve the components of the sound rendering computer 120.

[0039] The components (e.g., modules, processing unit 124) of the sound rendering computer 120 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some embodiments, the components of the sound rendering computer 120 can be configured to operate within a cluster of devices (e.g., a server farm). In such an implementation, the functionality and processing of the components of the sound rendering computer 120 can be distributed to several devices of the cluster of devices.

[0040] The components of the sound rendering computer 120 can be or can include any type of hardware and / or software configured to process attributes. Figure 1 One or more portions of the components shown in the components of the sound rendering computer 120 in the can be or can include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, a collection of computer-readable instructions that can be executed at a computer). For example, in some embodiments, one or more portions of the components of the sound rendering computer 120 can be or can include software modules configured for execution by at least one processor (not shown). In some embodiments, the functionality of the components can be included in conjunction with Figure 1 those shown in different modules and / or different components.

[0041] Although not shown, in some embodiments, the components of the sound rendering computer 120 (or a portion thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some embodiments, the components of the sound rendering computer 120 (or a portion thereof) can be configured to operate within a network. Therefore, the components of the sound rendering computer 120 (or a portion thereof) can be configured to function in various types of network environments that can include one or more devices and / or one or more server devices. For example, the network can be or can include a local area network (LAN), a wide area network (WAN), etc. The network can be or can include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network can include one or more segments and / or can have parts based on various protocols such as Internet Protocol (IP) and / or proprietary protocols. The network can include at least a portion of the Internet.

[0042] In some embodiments, one or more of the components of the sound rendering computer 120 can be or can include a processor configured to process instructions stored in a memory. For example, the sound acquisition manager 130 (and / or a portion thereof), the crosstalk cancellation manager 140 (and / or a portion thereof), and the VBAP manager 150 (and / or a portion thereof) can be a combination of a processor and a memory configured to execute instructions associated with a process for implementing one or more functions.

[0043] Figure 2 2 is a flow chart illustrating an example method 200 for mapping user interaction data to discrete buckets. The method 200 may be performed in conjunction with Figure 1The software constructs described together are executed, which reside in the memory 126 of the sound rendering computer 120 and are run by the processing unit set 124.

[0044] At 202 , the sound acquisition manager 130 receives audio data from an audio source at a source location, the audio data representing an audio waveform converted into sound of certain frequencies via a plurality of loudspeakers configured to be heard by a listener at a listener location, each of the plurality of loudspeakers having a corresponding loudspeaker location.

[0045] At 204 , the crosstalk cancellation manager 140 performs a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of respective audio signals emitted by the loudspeakers to determine spatialization cues in response to the frequency of the audio signal being below a specified threshold.

[0046] At 206 , the VBAP manager 150 performs VBAP operations on the plurality of loudspeakers to generate respective weights for the loudspeakers in response to the frequency of the audio signal being above a specified threshold, the respective weight for each of the plurality of loudspeakers representing a factor by which the audio signal emitted by the loudspeaker is multiplied to determine the spatialization cues.

[0047] Figure 3 is a schematic diagram illustrating an example geometry 300 used when considering crosstalk cancellation (CC) operation. Within geometry 300, a pair of loudspeakers 310(1) and 310(2) face a listener 320.

[0048] The propagation of sound from a source to a human listener is usually described in terms of a head-related transfer function (HRTF). An HRTF is a frequency response that describes the propagation from a point source at a specific location to the left and right ears in the absence of reverberation. An HRTF depends on many factors. For simplicity, it is usually simplified to depend on the source arrival orientation relative to the direction the head is pointing - i.e., azimuth and elevation. Other factors are usually ignored, such as distance, rotation of the head relative to the torso, etc.

[0049] The sound presented by the loudspeaker 310(1) is transmitted using (H 1L , H 1R ) is propagated to the two ears of the listener 320. Similarly, the sound presented by the loudspeaker 310(2) is presented as (H 2L , H 2R ) to the two ears of the listener 320. This means that - expressed in the frequency domain - the signals S1 and S2 played from the loudspeakers produce observed signals L and R that obey the following relationship: .

[0050] Assume that the desired binaural signal to be presented at both ears is represented by L des and R des Given, then this system of equations can be solved to obtain the appropriate S1 and S2 that, when played through a loudspeaker, will produce the desired signal at each ear: . Therefore, if the HRTF (H 1L , H 1R ) and (H 2L ,H 2R ) are known, one can generate the loudspeaker output signals necessary to deliver the spatialized audio to the listener 320.

[0051] Note that when the listener's position changes relative to the loudspeakers (or vice versa), the HRTF will change. Figure 4 An example of an HRTF that can change in real time as the listener moves is provided in .

[0052] Figure 4 Two source orientations (az, el) are shown, located respectively to the left and right of the listener's head: and The top row of panels shows the amplitude of the left and right ear transfer functions. The middle row of panels shows the amplitude of the left ear divided by the right ear frequency response. The bottom row shows the time propagation relative left to right ear time delay. These graphs show the following HRTF characteristics that are relevant for sound localization.

[0053] The interaural time difference (ITD) is the apparent relative delay in the source signal between the two ears. Figure 4 from the listener's left—that is, from —The source arriving first arrives at the left ear and secondly at the right ear. This produces a negative relative delay L / R (=ITD) observed for this source position. From the right side of the listener — i.e., from —The source of arrival shows the opposite behavior. The |ITD| of the more laterally located source is greater than that of the The |ITD| of the source of . The ITD is not constant with frequency, as it would be for a point in a free field. The presence of the head produces an ITD amplitude that is larger at lower frequencies than at higher frequencies.

[0054] The interaural level difference (ILD) is the relative level difference in the source signal between the two ears. Figure 4 The top and middle rows of the panels in the . From the listener's left—that is, from —The source arrives louder at the left ear than at the right ear because the head "masks" the source as it travels to the right ear. For this source position, this produces a proportional value of amplitude L / R (=ILD) expressed in dB. From the right side of the listener—that is, from —The source of arrival shows the opposite behavior. The |ILD| of a more laterally located source is usually greater than that of The |ILD| of the source is higher because of the higher degree of head shadowing. Similar to ITD, ILD is not constant with frequency. The presence of the head produces a larger ILD amplitude at higher frequencies than at lower frequencies.

[0055] As in Figure 4 The spectral clues to the distinct peaks, valleys and notches in the transfer function amplitude are shown in the top row of panels in . These arise from a variety of factors including ear canal resonance, reflections from the listener's torso / shoulders, and reflections from the outer ear or pinna.

[0056] Typically, interaural cues (ITD and ILD) reflect source lateralization (i.e., movement to the left or right side of the listener). The broad trends in ITD and ILD are similar across different listeners and even make themselves easy to simulate using a rigid ball head model. Since ITD begins to alias at higher frequencies, ITD is most relevant to lower frequencies (lower than ~1500 Hz). Mainly due to the reduced relevance of ITD at these frequencies, ILD is most relevant to higher frequencies (higher than ~1500 Hz).

[0057] When considered along a “cone of confusion” with similar lateralized source locations, interaural cues become ambiguous. , , and The sources at all locations are similarly directed along the spectral cues are often used by listeners to distinguish source locations along the same cone of confusion. In particular, spectral cues are useful for elevation localization and front / rear source discrimination. They are also useful for "externalization" - that is, making a sound appear as if it originates from an actual point outside the head. Spectral cues are highly individualized due to highly individual variations in the structure of the pinna across different listeners.

[0058] The telepresence system is configured to render the remote talker's speech as if the talker were in the listener's acoustic space. It is assumed that the sound rendering computer 120 has appropriately "cleaned up" the transmitted audio so that it is a single channel consisting of only the talker's speech. The task of the sound rendering computer 120 is to convert this single source into a binaural signal based on the relative positions and head orientations of the listener and talker. This is done by Figure 3 This is accomplished by applying the appropriate HRTF to the talker's voice as shown to produce the signal that should be presented to the listener's ears.

[0059] One technique for obtaining these signals is a rigid sphere model or a rigid sphere HRTF model for ILD / ITD rendering. Studies have shown that rigid sphere models are able to produce interaural cues, especially ITD, that mirror those observed with actual listeners. Figure 4 The synthetic HRTF based on the rigid sphere head model with a radius of 8.5 cm is also shown with a dotted line. (Other radii can be used, such as 8.0 cm, 9.0 cm, 7.5 cm, 9.5 cm, etc.) The interaural cues are very similar, but the high-frequency ILD tends to be reduced. There are no detailed spectral cues, but this is not unexpected. However, the rigid sphere model has the advantage of being fully parameterized and mathematically solvable.

[0060] Another technique that can be used is custom HRTF rendering, where the listener's own empirically derived HRTFs are applied. While this produces the most accurate and realistic binaural signals, in some implementations, the costs associated with this approach render it impractical as a general approach.

[0061] Another technique that can be used is reference set HRTF rendering. Rather than using the HRTF of an individual listener, an alternative would be to use a generic "typical" HRTF for spatialization, or an HRTF picked from a library of reference HRTFs. Since the interaural cues for ITD and ILD are usually similar across listeners, this will produce good spatialization, especially with respect to lateral sources.

[0062] Another technique that can be used is reference set ILD / ITD rendering. Instead of using full HRTFs to synthesize spatialization, a simpler alternative would be to synthesize only the interaural (ITD and ILD) localization cues. These cues are similar across listeners, so the use of interaural cues or a "reference set" will produce similar spatialization of lateral sources as achieved using the listener's own interaural cues. Furthermore, interaural cues are typically less "rich" than full HRTFs, which means that they can be parameterized or sampled in a less dense set of source orientations, thereby reducing memory usage at runtime.

[0063] As mentioned above, the CC operation described above is preferably performed for lower frequencies (eg, below between 1000 Hz and 2000 Hz). Above such frequencies, the improved technique includes performing a modified VBAP operation to produce a positive set of weights for at least some of the loudspeakers.

[0064] Figure 5is a schematic diagram illustrating an example geometry 500 used when considering a modified vector-based amplitude panning (VBAP) operation. In the geometry 500, there are four loudspeakers 510(1), 510(2), 510(3), and 510(4) aimed at a listener 530. There is also a virtual source 520 typically in front of the listener 530. The listener 530 is not necessarily equidistant from all of the loudspeakers 510(1-4) and can move around relative to them. In some embodiments, there are more than four loudspeakers near the listener 530. In some embodiments, there are two loudspeakers near the listener 530.

[0065] Figure 5 Shown are loudspeakers 510 (1-4) directed from the center of the listener 530 (or generally, the listener 530). and Virtual Source 520 From these unit vectors, the VBAP manager 150 generates an overdetermined (or underdetermined when the number of loudspeakers is less than three) linear system that produces weights corresponding to each of the loudspeakers 510 (1-4).

[0066] The solution of the linear system for conventional VBAP has several limitations. First, conventional VBAP assumes that the head of the listener 530 is positioned equidistant from all loudspeakers, e.g., 510(1-4). Second, conventional VBAP uses exactly three loudspeakers to spatialize the virtual source 520. When there are more than three loudspeakers, conventional VBAP requires the listener space to be divided into non-overlapping triangles so that each sub-area is covered by exactly three loudspeakers. In conventional VBAP, although spatialization is achieved by calculating VBAP weights for an appropriate subset of loudspeakers, it requires an arbitrary division of space into triangles. For example, when the loudspeakers 510(1-4) are arranged in a square centered on the listener, the square can be divided into two triangles in two different ways: 510(1,2,3)+510(2,3,4) or 510(1,2,4)+510(1,3,4); it is not clear which is preferable. Furthermore, the division into groups of three loudspeakers can lead to counterintuitive loudspeaker weightings. For example, consider the square geometry above divided into two triangular sub-regions spanned by 510(1,2,3)+510(2,3,4). In this case, a virtual source located exactly at the center of the square will only have non-zero VBAP weights for loudspeakers 510(2) and 510(3). A more intuitive VBAP weighting would have equal contributions from all four loudspeakers. Third, there is no guarantee that all weights found according to conventional VBAP will be all positive. Thus, the modified VBAP with respect to Figure 4 is presented.

[0067] Figure 6 is a flow chart illustrating an example method 600 for performing modified VBAP. The method 600 may be performed by Figure 1 The described software constructs are executed by the sound rendering computer 120 , which reside in the memory 126 and are executed by the processing unit set 124 .

[0068] At 602, the loudspeaker manager 152 generates a signal based on a unit vector To generate the loudspeaker matrix. Typically, the loudspeaker matrix has unit vectors in three dimensions for each column corresponding to each loudspeaker. For example, when there is When there are 10 loudspeakers, the loudspeaker matrix has the dimensions .for Figure 5 In the illustrated case, the matrix has dimensions . Therefore, the linear system is overdetermined.

[0069] At 604, the source vector manager 154 generates a source vector. In this case the source vector is simply a unit vector .

[0070] At 606, the pseudo-inverse manager 156 performs a pseudo-inverse operation on the loudspeaker matrix and the source vector to generate a weight vector. For example, in some embodiments, the pseudo-inverse manager 156 calculates the matrix To generate the loudspeaker matrix The Penrose pseudo-inverse of . In this case, the weights are then derived from the quantity The weight vector of the overdetermined system is not uniquely determined. In this case, the pseudo-inverse manager 156 generates a weight vector with the minimum norm , that is, the weight vector The sum of squares of the components is minimized.

[0071] At 608, the VBAP manager 150 determines whether all components of the weight vector are positive. If all weights are positive, the method 600 completes at 614. If not, the VBAP manager at 610 sets the weight vector All those components of are set to zero. In effect, the VBAP manager 150 removes those loudspeakers whose negative weights correspond to . In this case, at 612, the loudspeaker matrix manager 152 generates a new loudspeaker matrix , where the columns correspond to the negative weights that are removed. Method 600 then repeats until the weight vector until all components of are positive.

[0072] In some embodiments, after generating a weight vector with all positive components, the VBAP manager 150 multiplies each component by the corresponding head-to-loudspeaker distance. This multiplication corrects for the inverse square distance speaker energy loss due to waves propagating over different distances. In the absence of reverberation, this compensates for the non-reverberant path signal pointed at the loudspeaker for situations where the listener is at unequal distances from the loudspeaker. In some embodiments, the weight vector A phase component based on the distance between the listener and the loudspeaker may also be included. In this case, such a phase component aligns the phase of the signal at the listener's head.

[0073] The modified VBAP described above addresses all three of the above concerns. Specifically, (i) the modified VBAP does not assume that the listener is equidistant from all loudspeakers, (ii) the modified VBAP is applicable to 2+ loudspeakers, (iii) the subset of loudspeakers is selected through an iterative process rather than by arbitrarily predividing the space into triangles, (iv) for an arrangement such as a square, a source located at the center of the square receives equal VBAP contributions from all four vertex loudspeakers, and (v) all weights are positive.

[0074] The improved technique described above uses the tracked listener head position to continuously update the VBAP weights for correct source spatialization. Note that VBAP depends only on the listener head position and the virtual source positions. VBAP does not require knowledge of head rotation or HRTFs. This can result in less accurate spatialization cues than those provided by CC, but the spatialization cues are also less susceptible to tracking errors and HRTF inaccuracies.

[0075] In summary, CC requires knowledge of the listener position / rotation as well as the listener HRTF. VBAP, on the other hand, only requires knowledge of the listener position. In general, CC provides more accurate localization cues but is more sensitive to tracker (especially rotational) errors and is limited by the accuracy of the underlying HRTF model, while VBAP provides less accurate localization cues but is less sensitive to tracker errors and does not require HRTF knowledge at all. CC's sensitivity to tracker errors is wavelength-dependent—as the wavelength decreases, the tracker error becomes a larger fraction of the wavelength. In addition, the highly individual aspects of the listener HRTF are concentrated in high-frequency spectral cues that depend on the shape of the individual listener's outer ear (or pinna). Finally, sound localization (especially left / right localization) is dominated by low-frequency interaural cues.

[0076] These properties suggest a hybrid CC / VBAP approach that uses CC in the low frequency region and VBAP in the high frequency region. That way, the more accurate CC localization cues are maintained in the frequency region where they are most important and CC sensitivity to tracker errors and HRTF individualization is the lowest, and VBAP localization cues that are less sensitive to tracker errors are used elsewhere. Typical cutoff frequencies between the low and high frequency regions are in the range of 1000-2000 Hz (this reflects the fact that interaural time differences begin to alias spatially in this region).

[0077] Figure 7 An example of a general purpose computer device 700 and a general purpose mobile computing device 750 that may be used with the techniques described herein is illustrated.

[0078] like Figure 7 As shown in , computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the inventions described and / or claimed in this document.

[0079] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710 and 712 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 702 is capable of processing instructions for execution within the computing device 700, including instructions for displaying graphical information for a GUI on an external input / output device such as a display 716 coupled to the high-speed interface 708 stored in the memory 704 or on the storage device 706. In other embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 700 can be connected, where each device provides a portion of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).

[0080] The memory 704 stores information within the computing device 700. In one embodiment, the memory 704 is one or more volatile memory units. In another embodiment, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0081] The storage device 706 can provide mass storage for the computing device 700. In one implementation, the storage device 706 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine readable medium, such as a memory 704, a storage device 706, or a memory on a processor 702.

[0082] The high-speed controller manages bandwidth-intensive operations of the computing device 700, while the low-speed controller manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 710, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller is coupled to the storage device 706 and the low-speed expansion port. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0083] As shown in the figure, computing device 700 can be implemented in many different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown) (such as computing device 750). Each of such devices can contain one or more of computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0084] Computing device 750 includes processor 752, memory 764, input / output devices such as display 754, communication interface 766, and transceiver 768, among other components. Computing device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0085] Processor 752 is capable of executing instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. For example, the processor may provide for coordinating other components of computing device 750, such as control of a user interface, applications run by computing device 750, and wireless communications performed by computing device 750.

[0086] The processor 752 can communicate with the user through a control interface 758 and a display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 may receive commands from the user and convert them for submission to the processor 752. In addition, an external interface 762 in communication with the processor 752 may be provided so as to enable the computing device 750 to communicate with the near area of ​​other devices. The external interface 762 may, for example, be provided for wired communication in some embodiments, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0087] The memory 764 stores information within the computing device 750. The memory 764 can be implemented as one or more of a computer-readable medium or media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 can also be provided and connected to the computing device 750 through an expansion interface 772, which can include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 774 can provide additional storage space for the computing device 750, or can also store applications or other information for the computing device 750. Specifically, the expansion memory 774 can include instructions for executing or supplementing the above-mentioned processes, and can also include security information. Therefore, for example, the expansion memory 774 can be provided as a security module for the computing device 750, and can be programmed with instructions that allow the computing device 750 to be used securely. In addition, a security application can be provided via a SIMM card together with additional information, such as placing identification information on a SIMM card in an unbreakable manner.

[0088] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which can be received, for example, by transceiver 768 or external interface 762.

[0089] The computing device 750 can communicate wirelessly through a communication interface 766, which may include a digital signal processing circuit when necessary. The communication interface 766 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. Such communication may occur, for example, through a radio frequency transceiver 768. In addition, short-range communication may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 may provide additional navigation and location-related wireless data to the computing device 750, which may be used as appropriate by the application running on the computing device 750.

[0090] Computing device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it into usable digital information. Audio codec 760 may likewise generate audible sounds for the user, such as through a speaker in an earpiece of computing device 750, for example. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on computing device 750.

[0091] As shown in the figure, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile device.

[0092] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor that can be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0093] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0095] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a Web browser with which a user can interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0096] A computing system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0097] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0098] It should also be understood that when an element is referred to as being on another element, being connected to, being electrically connected to, being coupled to or being electrically coupled to another element, it can be directly on another element, being connected or being coupled to another element, or one or more intermediate elements can be present.On the contrary, when an element is referred to as being directly on another element, being directly connected to or being directly coupled to another element, there is no intermediate element.Although the term "directly on", "directly connected to" or "directly coupled to" may not be used in the entire detailed description, the element that is shown as being directly on, directly connected to or directly coupled can be referred to in this way.The claims of the present application can be modified to narrate the exemplary relationship described in the specification or shown in the drawings.

[0099] Although certain features of the described embodiments have been shown as described herein, those skilled in the art will now appreciate that many modifications, substitutions, changes and equivalents will be present. It should therefore be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It should be understood that they have been presented only as examples and not by way of limitation, and that various changes in form and detail may be made. Any part of the apparatus and / or method described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein can include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.

[0100] In addition, the logic flows depicted in the accompanying drawings do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be removed from the described flow, and other components can be added to or removed from the described system. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method comprising: receiving audio data from an audio source at a source location, the audio data representing an audio waveform that is converted to sound of a certain frequency via a plurality of loudspeakers configured to be heard by a listener at a listener location; generating a loudspeaker matrix having elements that are components of a vector parallel to a difference between the listener position and loudspeaker positions of the plurality of loudspeakers; generating a source vector having elements that are components of a vector parallel to a difference between the listener position and the source position; as well as A pseudo-inverse operation is performed on the loudspeaker matrix and the source vector to generate a weight vector having components.

2. The method according to claim 1, wherein: A distance between the listener and a first loudspeaker of the plurality of loudspeakers is different from a distance between the listener and a second loudspeaker of the plurality of loudspeakers.

3. The method according to claim 1, wherein: The number of loudspeakers in the plurality of loudspeakers is greater than three, and The performing of the pseudo-inverse operation on the microphone matrix and the source vector comprises generating a product of the microphone inverse and the Penrose pseudo-inverse of the source vector.

4. The method according to claim 3, wherein: Performing the pseudo-inverse operation on the loudspeaker matrix and the source vector further includes minimizing the sum of squares of components of the weight vector.

5. The method according to claim 3, wherein: The components of the weight vector are less than zero, and Wherein, the method further comprises: removing elements of the loudspeaker matrix corresponding to the loudspeakers corresponding to the components of the weight vector that are less than zero to form a reduced loudspeaker matrix; and The pseudo-inverse operation is performed on the reduced loudspeaker matrix and the source vector to produce a reduced weight vector.

6. The method according to claim 1, further comprising: The components of the weight vector are multiplied by a scaling factor that is proportional to a distance between the listener and the loudspeaker of the plurality of loudspeakers to which the components of the weight vector correspond.

7. The method according to any one of claims 1 to 6, wherein: Generating the loudspeaker matrix and the source vector is part of performing a vector-based amplitude panning (VBAP) operation on the plurality of loudspeakers, and Wherein, the method further comprises: In response to the frequency of the audio signal being below a specified threshold, performing a crosstalk cancellation (CC) operation on the plurality of loudspeakers to generate amplitudes and phases of corresponding audio signals emitted by the loudspeakers to determine spatialization cues; and In response to the frequency of the audio signal being above the specified threshold, the VBAP operation is performed on the plurality of loudspeakers to generate corresponding weights for the loudspeakers.

8. The method according to claim 7, wherein: Performing the CC operations on the plurality of loudspeakers includes tracking the position and orientation of the listener over time.

9. The method according to claim 7, wherein: The number of loudspeakers in the plurality of loudspeakers is an even number, and The performing the CC operation on the plurality of loudspeakers comprises: applying a head related transfer function HRTF configured to provide a binaural sound field to the listener to the corresponding loudspeakers, wherein the HRTF is based on a parameterized rigid sphere model.

10. A non-transitory computer-readable storage medium comprising code which, when executed by a processing circuit configured to perform audio source spatialization, causes the processing circuit to perform the method according to any one of claims 1 to 9.

Images