Boundary distance system and method
Through the time-frequency masking technology, the problem of automatic speaker calibration and arrival direction estimation under low signal-to-noise ratio conditions is solved, and the accurate speaker calibration and immersive sound generation in a noise environment is realized, which improves the robustness of the multi-speaker system Sex and audio quality.
Patent Information
- Application Number
- CN202410635324.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-22
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to achieve accurate automatic calibration and arrival direction estimation of speakers under low signal to noise ratio conditions, especially in noise environments, resulting in a degradation of the quality of immersive sound generation.
Using a time-frequency masking method, by using at least two microphones and speakers to play back the characteristic signals, combining generalized cross-correlation and feature frame detection, noise-rosted arrival direction estimation is achieved, noise-rosted influence and signal quality is improved.
Provide accurate direction of arrival estimation under low signal-to-noise ratio and reverb conditions, ensuring robustness of speaker automatic calibration and high audio quality for immersive sounds, suitable for multi-speaker configurations in complex environments.
Smart Images

Figure CN119071687A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is generally related to U.S. application serial number 18 / 204,150, entitled “SYSTEM AND / OR METHOD FOR LOUDSPEAKER AUTO CALIBRATION AND LOUDSPEAKER CONFIGURATION LAYOUT ESTIMATION”, filed on May 31, 2023, and having agent case number P220102US (HARM0866PUS), the disclosure of which is hereby incorporated by reference in its entirety.
[0003] This application is generally related to U.S. application serial number 18 / 204,159, entitled “APPARATUS, SYSTEM AND / ORMETHOD FOR NOISE TIME-FREQUENCY MASKING BASEDDIRECTION OF ARRIVAL ESTIMATIONFOR LOUDSPEAKER AUDIO CALIBRATION,” filed on May 31, 2023, and agent case number P220104US (HARM0867PUS), the disclosure of which is hereby incorporated by reference in its entirety. Technical Field
[0004] Aspects disclosed herein generally relate to a noise-robust time-frequency masking-based direction-of-arrival estimation device, system, and / or method for speaker audio calibration. These and other aspects will be discussed in greater detail herein. Background Art
[0005] Various speaker manufacturers or providers may bring together various speaker categories to form an ecosystem. In this regard, the various speakers may communicate or work with each other and / or with a mobile device. Thus, such speakers may use immersive sound to achieve higher audio quality. For immersive sound generation, information related to the location of the speaker may be required. Therefore, automatic calibration may be required before the speaker can generate immersive sound. Summary of the invention
[0006] In at least one embodiment, an audio system is provided that includes a speaker, at least one microphone, and at least one controller. The speaker transmits an audio signal into a listening environment defined by at least one wall in a room. The at least one microphone is positioned on the speaker and is configured to capture a reverberant audio signal including a plurality of reverberations and a plurality of peaks. The reverberant audio signal indicates that the audio signal is reflected from the at least one wall. The at least one controller is programmed to apply a confidence score to the plurality of peaks to obtain a maximum score indicating a maximum peak of the audio signal reflected from the at least one wall, and to determine a distance between the speaker and the wall based at least on the maximum score.
[0007] In at least another embodiment, a method is included. The method includes transmitting an audio signal via a speaker into a listening environment defined by at least one wall in a room and capturing via at least one microphone a reverberant audio signal including a plurality of reverberations, wherein the reverberant audio signal indicates that the audio signal is reflected from the at least one wall. The method also includes applying a confidence score to a plurality of peaks to obtain a maximum score indicating a maximum peak of the audio signal reflected from the at least one wall and determining a distance between the speaker and the wall based at least on the maximum score.
[0008] In at least another embodiment, a computer program product embodied in a non-transitory computer-readable medium stored in a memory is provided, the computer program product being programmed and executable by at least one controller in an audio system. The computer program product includes instructions to transmit an audio signal via a speaker into a listening environment defined by at least one vertical surface in a room and capture a reverberant audio signal including a plurality of reverberations, wherein the reverberant audio signal indicates that the audio signal is reflected from the at least one vertical surface. The computer program product also includes instructions to apply confidence scores to a plurality of peaks to obtain a maximum score indicating a maximum peak of the audio signal reflected from the at least one vertical surface, and to determine a distance between the speaker and the at least one vertical surface based at least on the maximum score. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Embodiments of the present disclosure are particularly pointed out in the appended claims. However, other features of the various embodiments will become more apparent and will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which:
[0010] Figure 1 A system for performing noise-robust time-frequency masking-based directional and speaker automatic calibration and speaker configuration layout estimation according to one embodiment is generally described;
[0011] Figure 2 Depicting how to remove noise by applying time-frequency (TF) masking according to one embodiment;
[0012] Figure 3 depicts the output of a TF masking block according to one embodiment;
[0013] Figure 4 depicts two-microphone direction of arrival estimation using time difference of arrival;
[0014] Figure 5 depicts various characteristic tone signals according to one embodiment;
[0015] Figure 6 Described is a method for performing optimized speaker auto-calibration and speaker configuration estimation according to one embodiment;
[0016] Figure 7 depicts an example of a speaker and microphone configuration according to one embodiment;
[0017] Figures 8 to 9 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of microphone orientation and speaker placement estimation operation of the method;
[0018] Figures 10 to 12 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of outlier detection operation of the method;
[0019] Fig.13 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of a reference speaker selection operation of the method;
[0020] Fig.14 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of microphone orientation and speaker placement estimation operation of the method;
[0021] Fig.15 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of microphone orientation and speaker placement estimation operation of the method;
[0022] Fig.16 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6Another example of a candidate coordinate estimation operation of the method;
[0023] Fig.17 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0024] Fig.18 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0025] Fig.19 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0026] Fig. 20 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0027] Fig.21 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of the optimal coordinate selection operation of the method;
[0028] Fig. 22 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 Another example of the optimal coordinate selection operation of the method;
[0029] Fig.23 depicts an example of microphone orientation and speaker estimation according to one embodiment;
[0030] Fig.24 A system for performing boundary estimation according to one embodiment is depicted;
[0031] Fig.25 depicts an example of a frequency response to an exponential sine sweep (ESS) for stimulating a listening environment according to one embodiment;
[0032] Fig.26 depicts an example of an amplitude spectrum of an inverse filter according to one embodiment;
[0033] Fig. 27 depicts an example of a room impulse response (RIR) measurement according to an embodiment;
[0034] Fig.28 depicts an example of peak detection involving RIR measurements according to one embodiment; and
[0035] Fig.29 A method for performing boundary estimation involving multiple speakers is described according to one embodiment. DETAILED DESCRIPTION
[0036] As required, detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary embodiments of the present invention that may be embodied in multiple and alternative forms. The drawings are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural details and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the present invention in various ways.
[0037] Noise-robust time-frequency masking based direction-of-arrival estimation for loudspeaker automatic calibration
[0038] Speakers are used to produce immersive sound effects. One aspect of immersive sound is that automatic calibration is required to locate the position of the speaker. A method for performing speaker positioning includes estimating the azimuth of the speaker, which is also called direction of arrival estimation (DOA). For low signal-to-noise ratio (SNR) (i.e., less than 0dB), the performance of the DOA method may be problematic because noise is the main signal under low SNR conditions. In addition, in real scenarios, noise may not be avoided for the automatic calibration stage. Therefore, for the automatic calibration stage, a noise-robust DOA estimation method is required. The disclosed system and / or method utilizes time-frequency (TF) masking that can be used for source separation as a preprocessing step for the DOA estimation method to achieve high performance under low SNR. TF masking can extract the desired signal from the noise signal captured by the microphone. The various aspects provided herein also provide a characteristic signal that maximizes performance under low SNR conditions. The embodiments disclosed herein provide DOA estimation based on TF masking designed using at least two microphones and a characteristic signal that can be played back by a speaker.
[0039] As noted above, for immersive sound generation of loudspeakers, automatic calibration is typically required. Failure in the automatic calibration phase may result in negative feedback from the listener. In addition, since the environment cannot be controlled during the automatic calibration phase, background noise is not avoidable. Therefore, for immersive sound generation using multiple loudspeakers, noise-robust automatic calibration is desired. The disclosed embodiments provide noise-robust automatic calibration to provide immersive sound generation. In addition, the disclosed system typically provides accurate DOA estimates for loudspeaker automatic calibration under low signal-to-noise ratios and reverberation. These aspects enable immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner cases (such as two loudspeakers located, for example, on the same line but not aimed at each other).
[0040] One way of automatic calibration of a speaker may involve estimating the angle of the speaker, also known as DOA. There are many techniques for estimating the DOA of a talker / speaker, such as time difference of arrival (TDOA), multiple signal classification (MUSIC), and controlled response power (SRP). While the TDOA method does not provide satisfactory performance at low signal-to-noise ratios (SNRs), MUSIC and SRP require a large number of microphones for high performance at low SNRs. Even the MUSIC method and the SRP method perform below requirements under low SNR conditions (i.e., -10 dB buzzing noise). The disclosed system provides a characteristic tone in the form of an inverse exponential sine sweep (ESS) signal, which has been found to provide, among other things, an indication to a controller to initiate automatic calibration of the speaker in a noisy environment (such as -10 dB). In these types of noisy environments, other types of characteristic tones that do not utilize ESS-based signals may not be perceptible to the controller.
[0041] Figure 1 A system 100 for performing noise robust time-frequency masking-based directional and speaker automatic calibration and speaker configuration layout estimation according to one embodiment is depicted. The system 100 includes a speaker 102 having a plurality of microphones 106a to 106b (or "106"), a time-frequency (TF) masking block 108, a feature frame detection block 110, a generalized cross-correlation (GCC) phase transform (PHAT) block 112, at least one controller 122 ("controller 122"), and a memory 130. The system 100 also includes a mobile device 150 having a matrix array 114, a microphone orientation estimation (MOE) block 116, an outlier detection block 118, an optimization block 120, and at least one controller 123 (hereinafter "controller 123"). It should be appreciated that the controller 123 may execute any instructions for any function performed by the mobile device 150 as described herein. Although Figure 1A single speaker 102 is illustrated, but it should be appreciated that the system 100 includes any number of speakers 102 positioned therein.
[0042] At least one of the speakers 102 transmits an audio signal including the characteristic sound 104 into the listening environment 151 to the other speakers 102 in the system 100. It should be appreciated that the speakers 102 typically include at least two of the microphones 106a to 106b. The speakers 102 can transmit an audio signal including the characteristic sound 104 into the listening environment 151. The microphones 106a to 106b positioned on different speakers 102 capture the audio signal including the characteristic sound 104. Each speaker 102a and 102b includes a memory 130. The memory 130 of the speaker 102b stores the audio signal and the corresponding characteristic sound (or characteristic frame) 104 for processing.
[0043] As noted above, the TF masking block 108, the feature frame detection 110, the GCC PHAT block 112, and the controller 122 are implemented in all speakers 102 present in the system 100. Assuming, for example, that the system 100 includes four speakers 102, the first speaker 102 receives audio signals and corresponding feature sounds 104 from the other speakers 102. Therefore, in this regard, each speaker 102 estimates the direction of arrival (DOA) of the audio signals received from the other three speakers 102. The mobile device 150 includes one or more transceivers 155 to wirelessly receive the DOA estimates from each of the speakers 102 in the system 100. It should also be appreciated that each of the speakers 102 in the system 100 may also include one or more transceivers 152 to wirelessly transmit the estimated DOA information to the mobile device 150.
[0044] In general, the TF masking block 108 in the speaker 102 reduces the noise impact associated with the captured audio signal received from other speakers 102 in the system 100. For example, the controller 122 applies the TF masking block 108 to each microphone input to reduce the noise impact. After the TF masking block 108 reduces the noise impact, the feature frame detection block 110 estimates the feature sound 104. In one example, the length of the feature sound 104 can be 200msec. However, since the speaker 102 does not know when the feature sound 104 is played by another speaker 102, the speaker 102 records the received audio, for example, for more than 200msec. It can be assumed that the speaker 102 can be in a recording mode while another speaker 102 transmits the feature sound 104. It is generally expected to detect a sufficiently long duration of the feature sound 104 to correctly estimate the DOA. The reception of the feature sound 104 on the audio signal can indicate to the receiving speaker 102 that the system 100 may be in an automatic calibration mode. In the automatic calibration mode, the speakers 102 may transmit information corresponding to the positions of the speakers 102 relative to the mobile device 150 (or other audio source).
[0045] The controller 122 applies a cross-correlation between the characteristic sound 104 played by the transmitting speaker 102 and the acquired audio. The cross-correlation performed by the GCC PHAT block 112 provides the location of the characteristic sound 104 in the long recording. In this regard, the controller 122 uses this location to extract the characteristic sound 104. At this point, the extracted characteristic sound 104 is provided to the GCC-PHAT block 112. The controller 122 can then use the estimated DOA to perform automatic calibration of the speaker 102b. These aspects will be discussed in more detail below. Referring back to the TF masking block 108, the controller 122 applies a TF masking operation as a preprocessing step for DOA estimation. The TF masking block 108 can eliminate the TF window (bin) dominated by noise in the audio signal to minimize the effects of noise and reverberation. Combined with Figure 2 A noisy input audio signal including a characteristic tone 104 is generally shown at 200. Figure 2 As shown, the noisy input audio signal including the characteristic tone 104 includes a noise sweep sine between 6 kHz and 7 kHz. Figure 2 In response to the TF masking block 108 performing a TF masking operation, the controller 122 extracts the characteristic tone 104 or signal from the noise mixture of the input audio signal, as generally shown at 202 in FIG.
[0046] Return to reference Figure 1 TF masking block 108 uses source separation and speech enhancement techniques. TF masking block 108 receives characteristic sound 104 to generate an enhanced signal. Controller 122 generates sampling delays using characteristic sound 104 via the enhanced signal. The controller 122 uses the sampling delay To determine the DOA of the received audio signal at the receiving speaker 102. As noted above, TF-based masking techniques may include ideal binary masking (IBM), ideal ratio masking (IRM), complex ideal ratio masking (cIRM), optimal ratio masking (ORM), etc. In general, when TF-based DOA estimation is employed, such masking techniques should not modify the phase information. Based on various requirements and tests, the disclosed system 100 may employ an IRM defined as follows:
[0047]
[0048] References to equation 1 can be found, for example, in: "The Optimal Ratio Time-Frequency Mask for Speech Separation in Terms of Signal-to-Noise Ratio" The Journal of the Acoustical Society of America 134, No. 5 (2013): EL452-EL458. When S(t, f) is the frequency response of the characteristic signal (or characteristic tone 104), N(t, f) represents the noise spectrum and β is a smoothing factor. Due to the comprehensive knowledge of the characteristic tone 104, S(t, f) can be calculated. The denominator in equation (1) can be the signal captured at microphones 106a to 106b. After the controller 122 calculates the mask, the enhanced signal can be calculated using the multiplication of the captured signal and the mask as in equation (2).
[0049] E(t,f)=IRM(t,f)*Y(t,f) (2)
[0050] E(t,f) represents an enhanced signal, which is a two-channel signal because the two microphones 106a and 106b of the receiving speaker 102 each receive an incoming audio signal including the characteristic sound 104. Y(t,f) corresponds to the signal captured at the microphones 106a to 106b. The enhanced signal may correspond to Figure 2 A signal is shown generally at 202 in which noise is removed from the captured audio signal.
[0051] Figure 3An example of a recorded signal 300 provided by a TF masking block 108 of a speaker 102 according to one embodiment is depicted. For example, the recorded signal 300 corresponds to the output provided by the TF masking block 108 after the TF masking block 108 performs a masking operation. In general, the recorded signal 300 provided by the TF masking block 108 may include a long sequence of audio data 302 including silence / noise and characteristic sounds 104. The characteristic sounds 104 are generally bounded by frames 304. As shown in FIG. Figure 3 The two audio signals shown correspond to one audio signal received at microphone 106a and another audio signal received at microphone 106b, which are then processed by TF masking block 108. Controller 122 detects frame 304 using cross-correlation between audio data 302 and feature sounds 104. In general, controller 122 detects an enhanced version of frame 304 (e.g., feature sounds 104 and acquired audio signals (e.g., which correspond to recorded signal 300, audio data 302, and frame 304)) to detect the start time of frame 304. GCC PHAT block 112 then receives frame 304 as input.
[0052] refer to Figure 1 and Figure 4 , the GCC PHAT block 112 processes the output of the feature frame detection block 110 to provide an estimated DOA for the captured audio signal transmitted by at least the speaker 102. An example of the operation of the GCC PHAT is described in the following: "The Generalized Correlation Method for Estimation of Time Delay", IEEE transactions on acoustics, speech, and signal processing 24, No. 4, (1976): 320-327, which is incorporated herein by reference in its entirety. As noted above, the speaker 102 in the system 100 can provide (or wirelessly transmit) the estimated DOA reading to the mobile device 150 (see Figure 1 ). The mobile device 150 may be a cell phone, a laptop computer, a desktop computer, etc. Also as noted above, the speakers 102 in the system 100 may include one or more transceivers 152 for wirelessly transmitting and receiving information (including estimated DOA readings) to each other and / or to the mobile device 150.
[0053] The GCC PHAT block 112 can utilize single path wave propagation of sound waves from a single sound source signal s(n) provided by a sound source (or any one of the speakers 102). The microphone 106a and the microphone 106b receive the signal () as the received signal x 1(n) and x 2 (n), the received signal is a delayed and attenuated version of the original sound signal (). In general, the controller 122 can obtain x 1 (n) and x 2 The received signal x is determined by the maximum value of the cross-correlation of 1 (n) and x 2 (n) The controller 112 performs the cross-correlation by executing the following equation:
[0054]
[0055]
[0056]
[0057]
[0058]
[0059] The sample delay is estimated using equations 3 to 7 in the GCC PHAT block 112 Equation 3 represents x 1 (n) and x 2 Equation 4 is the cross power density, which is obtained by taking x 1 (n) and x 2 The inverse Fourier transform is applied to obtain x 1 (n) and x 2 (n) is shown in Equation 6. Finally, by solving for x in Equation 7 1 (n) and x 2 (n) to calculate the sampling delay
[0060] At this time, the controller 122 can determine the DOA of the received audio signal or the angle of the sound source 102a (or the first speaker 102a). For example, the controller 122 can determine the DOA (or angle information, "angle") of the audio signal received by the receiving speaker 102 by:
[0061]
[0062] in is an estimate of the sample delay as noted above, c is the speed of sound, and d is the distance between microphone 106a and microphone 106b, which is a known value. GCC Phat block 112 estimates the phase difference between the audio captured between microphone 106a and microphone 106b. Thus, the phase difference generally corresponds to (or angle information), as shown in Equation 8. The controller 122 converts the phase difference into enable using inverse cosine or the like, as shown in Equation 8. Figure 4 The controller 122 determines the sampling delay way.
[0063] Figure 5 Various characteristic tone signals 500, 502 are depicted according to one embodiment. In general, the characteristic tone 500 includes energy below 4kHz. The characteristic tone 500 can be generated based on an exponential sine sweep (ESS). In this case, it may be more desirable to provide a characteristic signal that includes more high frequency energy for performing the estimated DOA in a higher noise based environment. The characteristic tone signal 502 is generated based on an inverse ESS. Figure 5 The two characteristic sound signals 500 and 502 are illustrated to be both in the frequency domain. The characteristic sound signal 502 has a higher amplitude after 1kHz, which prevents the characteristic sound signal 502 from being distorted by background noise. In one example, the characteristic sound 104, as generated by the first speaker 102a or the second speaker 102b, can be based on an inverse ESS of a predetermined frequency range that can be 700Hz to 10kHz, and has a predetermined length of, for example, 150ms at a predetermined frequency of, for example, 48kHz. The disclosed system 100 generally provides accurate DOA estimation for speaker automatic calibration under low signal-to-noise ratio and reverberation. These aspects enable immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner cases (such as two speakers located on, for example, the same line but not aimed at each other). It should be appreciated that it has been found that the characteristic sound signals 500 and 502 (e.g., inverse EES signals) can withstand high noise environments at least -10dB level. For example, it has been found that the inverse ESS signal is not affected in high noise environments of at least −10 dB, which helps provide a sufficient signal to trigger automatic calibration and determination of the DOA of each speaker 102 in the system 100 .
[0064] Improvements to automatic speaker calibration and speaker configuration layout estimation
[0065] As noted above, the speakers 102 in the system 100 are configured to communicate with each other. Each of the first speaker 102a and the second speaker 102b can provide high audio quality while utilizing immersive sound. The immersive sound technology depends on the positions of the first speaker 102a and the second speaker 102b. Therefore, in this regard, the immersive sound technology requires an automatic speaker calibration process.
[0066] There are many ways to perform automatic calibration. One way to perform automatic calibration requires providing an estimate of the azimuth angle of the speaker, also referred to as DOA. The DOA of the audio signal transmitted from each speaker can be detected by playing a characteristic tone from one speaker at a time. The angles (or DOA) from different speakers are then used to create the speaker configuration in the room. In some cases, due to environmental conditions and the position of the speakers, there may be errors in obtaining an estimate of the azimuth angle. Such errors may mainly occur when the speakers are not aimed at each other (for example, due to speaker directivity) and the background noise has more energy than the characteristic tone. Since these aspects may occur in real-world situations, the automatic calibration technology implemented in the speaker should address these situations. The system 100 disclosed herein provides multiple DOA estimates for optimizing speaker positions and estimating speaker layout configurations of two or more speakers. The system 100 also provides an accurate representation of the speaker configuration required for a truly immersive experience. The disclosed embodiments can improve robustness and overcome the above-mentioned environmental conditions. Furthermore, the disclosed embodiments can provide (i) accurate speaker configuration estimation, (ii) speaker orientation estimation, (iii) detection of DOA estimate outliers while taking into account background noise, reverberation, and obstacles, and (iv) optimizing speaker configuration estimation based on previous DOA estimates and outlier detection.
[0067] Return to reference Figure 1 , the system 100 also includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. The matrix block 114 stores the DOA estimation for each of the first speaker 102a and the second speaker 102b. As noted above, it should be appreciated that the system 100 may include any number of speakers 102, however, preferably, more than two speakers 102 may be required. The microphone orientation estimation block 116 estimates the orientation of each microphone in the microphones 106a and 106b as positioned on the speaker 102. The outlier detection block 118 detects outliers that may exist in the matrix formed by the matrix block 114. These outliers or errors may be attributed to an erroneous DOA estimation or an obstacle between the first speaker 102a and the second speaker 102b. The optimization block 120 performs reference microphone selection, initial layout estimation, candidate coordinate estimation, and optimal coordinate selection. These aspects will be discussed in more detail below.
[0068] Figure 6 Depicted is a method 600 for performing optimized speaker auto-calibration and speaker configuration estimation according to one embodiment.
[0069] In operation 602, the microphone orientation estimation block 116 estimates the orientations of the microphones 106a and 106b. Figure 8 and Fig. 9 This operation is discussed in more detail.
[0070] In operation 604, the outlier detection block 118 detects outliers with respect to DOA that may exist in the matrix formed by the matrix block 114. Figure 10 to Figure 11 This operation is discussed in more detail.
[0071] In operation 606, the optimization block 120 performs reference microphone selection. Fig.12 This operation is discussed in more detail.
[0072] In operation 608, the optimization block 120 uses the DOA estimate to perform an initial layout estimate. Fig.13 This operation is discussed in more detail.
[0073] In operation 610, the optimization block 120 calculates candidate coordinate estimates. Figures 14 to 16 This operation is discussed in more detail.
[0074] In operation 612, the optimization block 120 selects the best coordinates. Figure 17 to Figure 18 This operation is discussed in more detail.
[0075] Figure 7 An example of a speaker and microphone configuration 700 in system 100 according to one embodiment is depicted. Configuration 700 includes Figure 1 The speaker 102. Reference Figure 7 and below, Figure 1 The speakers 102 are generally shown as a first speaker 102a, a second speaker 102b, a third speaker 102c and a fourth speaker 102d. Figure 1As indicated, any number of speakers may be provided. Each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a first microphone 106a and a second microphone 106b. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a controller 122, a memory 130, and a transceiver 152. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a TF masking block 108, a feature frame detection block 110, and a GCC PHAT block 112. Also as indicated above, the mobile device 150 includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. It should be appreciated that in another embodiment, the system 100 may include a main speaker 103. The primary speaker 103 may correspond to any of the speakers 102a to 102d and may simply be designated as the primary speaker to perform similar tasks as performed by the mobile device 150. For example, the primary speaker 103 may be arranged to provide a layout of the speakers 102, including the layout of the primary speaker 103, based on the principles disclosed herein. In this sense, if it is preferred that the primary speaker 103 can provide the positions of the respective speakers 102 and 103 in the listening environment 151 so as to establish channel allocations for the speakers 102 and 103, then the primary speaker 103 provides a level of functionality similar to that provided in conjunction with the mobile device 150. Thus, in this regard, the primary speaker 103 may include a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. While the primary speaker 103 may provide the locations of the speakers 102 , 103 in the listening environment 151 in a manner similar to that explained for the mobile device 150 , the primary speaker 103 may not provide any visual indication or prompts to the user regarding the locations of the speakers 102 , 103 .
[0076] The first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d wirelessly communicate with each other and / or with the mobile device 150 via the transceiver 152 to provide a layout of speakers in the listening environment 151. In particular, the mobile device 150 can provide a layout of the respective speakers 102a, 102b, 102c, and 102d as arranged in the listening environment 151. In general, the specific layout of the speakers 102a-102d may be unknown to each other, and the various aspects described herein can determine the specific layout of the speakers 102a-102d in the listening environment 151. Once the layout of the speakers 102a-102d is known, the mobile device 150 can assign channels to the speakers 102a-102d in a deterministic manner based on a pre-stored or predetermined system configuration.
[0077] The mobile device 150 may display a layout of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d based on information received from such devices. In one example, the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d may wirelessly transmit the following to each other and / or to the mobile device 150 via the transceiver 152: DOA estimates, microphone orientation estimation information, outlier information, reference speaker selection information, initial speaker layout estimates, candidate coordinate estimation information, and optimal coordinate selection information as set forth in the method 600.
[0078] A legend 702 is provided, which illustrates various position angles of microphones 106a to 106b on one speaker 102 relative to microphones 106a to 106b on other speakers 102a, 102b, 102c, and 102d. The legend 702 will be referenced when describing the various operations of the method 600 below. The first speaker 102a, the third speaker 102c, and the fourth speaker 102d illustrate that their corresponding microphones 106a to 106b are horizontally arranged on such speakers 102a, 102c, and 102d. The second speaker 102b illustrates that the microphones 106a to 106b are vertically arranged on the second speaker 102b. It should be recognized that the arrangement of the microphones 106a to 106b is not known before determining the speaker layout, and the arrangement of the microphones 106a to 106b can be arranged in any number of configurations on the speakers 102a to 102d in the listening environment 151. The disclosed system 100 and method 600 are configured to determine a speaker configuration layout while taking into account different configurations of the microphones 106a-106b.
[0079] Referring to the first speaker 102a and further referring to the legend 702, the first speaker 102a is capturing audio (or detecting audio) from the second speaker 102b at 0 degrees. The first speaker 102a is capturing audio (or detecting audio) from the third speaker 102c at 45 degrees. The first speaker 102a is capturing audio from the fourth speaker 102d at an angle of 90 degrees. Figure 7 , the angles (or angle information) at which the remaining speakers 102b to 102d receive audio relative to the other speakers 102a to 102d are illustrated. Any reference to the term "angle" may also correspond to "angle information" and vice versa. The relevance of angles (or angle information) will be discussed in more detail below. It should be appreciated that each of the speakers 102a to 102d transmits information related to the angle information at which they receive audio from each other to the mobile device 150 or other suitable computing device. The mobile device 150 stores the angles in its memory. The DOA information as reported by the speakers 102a to 102d is reported as angles as mentioned above.
[0080] Figures 8 to 9 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 600 and an example of microphone orientation and operation 604 of the method 600. At least one embodiment provides a dual speaker position in space that establishes a line, and the slope of the line does not change when viewed from one speaker or the other. In general, the system 100 and / or method 600 identifies that the dual speaker position in space establishes a line, and the slope of the line does not change from one speaker to another. A first matrix 800 is illustrated, which depicts the relative angles of the audio received relative to each speaker 102a to 102d (or S1 to S4, respectively). Any angle reading of -360 represents a null value. For example, the first matrix 800 illustrates that S1 in both the columns and rows of the matrix is -360 because the first speaker 102a (or S1) cannot receive audio from itself. This is further illustrated for any value of the angle of 360 illustrated for the second speaker 102b (or S2), the third speaker 102c (or S3), and the fourth speaker 102d (or S4).
[0081] The mobile device 150 generally stores information corresponding to the angle information depicted by the first matrix 800. The first column as shown by the dotted box illustrated by the first matrix 800 corresponds to a specific speaker that receives audio from the speakers S1 to S4 as illustrated in the 2nd to 5th columns, respectively. For example, referring to the first column and the second row, the second speaker (e.g., or S2) 102b receives audio from the first speaker (e.g., or S1) 102a (as shown in the second column) at an angle of 90 degrees, the second speaker 102b receives audio from the third speaker 102c at an angle of 0 degrees, and the second speaker 102b receives audio from the fourth speaker 102d (or S4) at an angle of 45 degrees. Referring to the first column and the third row, the third speaker 102c (e.g., or S3) receives audio from the first speaker 102a (e.g., or S1) at an angle of 45 degrees, and the third speaker 102c (e.g., or S3) receives audio from the fourth speaker 102d (e.g., or S4) at an angle of 0 degrees. Referring to the first column and the fourth row, the fourth speaker 102d receives audio from the first speaker 102a (or S1) at an angle of 90 degrees, the fourth speaker 102d receives audio from the second speaker 102b (or S2) at an angle of 135 degrees, and the fourth speaker 102d receives audio from the third speaker 102c (or S3) at an angle of 180 degrees.
[0082] refer to Fig. 9 , the mobile device 150 receives information corresponding to each angle from the transceiver 152 of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. As noted above, the mobile device 150 assembles the first matrix 800 based on the information received from the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. The mobile device 150 can determine the orientations of the microphones 106a to 106b of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to each other. In particular, the mobile device 150 can determine whether the orientations of the microphones 106a to 106b of the respective first speakers 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are different from each other based on the angles stored in the first matrix 800. The embodiments disclosed herein generally illustrate that the slope may not change, but the angle depends on the orientation of the microphones 106a - 106b , which may be from the first row and first column, as generally shown at 900 .
[0083] For example, the mobile device 150 may determine whether the difference between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d as illustrated in the first matrix 800 corresponds to one or more predetermined values (e.g., 0 or 180). If the difference between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d corresponds to one or more predetermined values, the mobile device 150 may determine that the microphones 106a-106b of the two or more speakers 102a, 102b, 102c, 102d are in the same orientation. If the difference between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d does not correspond to one or more predetermined values, the mobile device 150 may determine that the microphones 106a-106b are not in the same orientation for the two or more speakers 102a, 102b, 102c, 102d.
[0084] References Fig. 9 In the illustrated first matrix 800, the mobile device 150 determines that the second speaker 102b has a 90 degree difference relative to the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. Therefore, in this regard, the mobile device 150 determines that the orientation of the microphones 106a-106b of the second speaker 102b is different from the orientation of the microphones 106a-106b of the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. This situation Fig. 9 Shown in.
[0085] Generally, the mobile device 150 performs microphone orientation estimation by subtracting the angle from the first column from the first row. The subtraction operation is then performed, and the result is [0,90,0,0] for the first speaker 102a (or S1), the second speaker 102b (or S2), the third speaker 102c (or S3), and the fourth speaker 102d (or S4). Therefore, the microphone estimation for the third speaker 102c (S3) and the fourth speaker 102d (S4) is 0, which is the same orientation as the orientation of the first speaker 102. Since the angle range should be [0,180], as Figure 7, so the mobile device 150 can also perform microphone orientation using a modular operation after performing a subtraction operation. In general, the slope between the speakers 102a to 102d may not change, but the angle may depend on the orientation of the microphones 106a to 106b. The mobile device 150 generates a microphone orientation array 900, which includes angle differences that do not correspond to 0 degrees or 180 degrees (or a predetermined value). The embodiments disclosed herein generally illustrate that the slope may not change, but the angle depends on the orientation of the microphones 106a to 106b, which can be found from the first row and first column of the microphone orientation array.
[0086] Figures 10 to 12 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 6. After performing the microphone orientation estimation of operation 602, the mobile device 150 performs the outlier detection operation 604 to determine whether any one of the speakers 102a to 102d is an outlier relative to the layout. If any one of the speakers 102a to 102d is determined to be an outlier, the mobile device 150 determines that the position of the speaker 102 is incorrect or cannot be determined relative to the positions of the other speakers 102a to 102d.
[0087] refer to Fig.10 As part of operation 604, the mobile device 150 subtracts the microphone orientation array 900 from the first matrix 800 to provide a calibrated angle matrix 902. The mobile device 150 considers a speaker(s) having a microphone orientation that is not aligned with the rest of the speaker's microphones. Fig.10 The calibrated angle matrix 902 is shown, showing that the 90 degree angles with the microphone orientation array 900 are subtracted from the angles in the second row (S2) (except -360 as this is a null value) to provide the following in the calibrated angle matrix 902: [0, -360, 90, 135].
[0088] refer to Fig.11 According to one example, as part of operation 604, the mobile device 150 may compare the angles shown in row 2 of the calibrated angle matrix 902 to a predetermined threshold. If any one or more of the angles in row 2 of the calibrated angle matrix 902 is above the predetermined threshold, the mobile device 150 detects an outlier for one or more speakers 102a to 102d having angles above the predetermined threshold. The mobile device 150 generates a Fig.11The block matrix 1100 is generally shown. For example, the mobile device 150 checks the difference between each pair of estimates (e.g., the angle estimate of the second speaker 102b at the first speaker 102a and the angle estimate of the first speaker 102a at the second speaker 102b). The mobile device 150 can apply, for example, modulus 180 to ensure that the difference is within the range of [0,180] degrees. If the difference is above a predetermined threshold, the mobile device 150 can determine that there is an outlier for the pair of speakers. In this example, the block matrix 1100 does not indicate an error for any of the speakers 102a to 102d. Therefore, at this point, the angle in the second row of the calibrated angle matrix 902 is less than the predetermined threshold. Outliers generally indicate various erroneous estimates in the DOA matrix. Detected outliers may not be used for optimization (e.g., operations 606, 608, and 610).
[0089] refer to Fig.12 And combined with Fig.11 Similar to the example noted above, according to another example, as part of operation 604, the mobile device 150 may compare the angle shown in row 2 of the calibrated angle matrix 902 with a predetermined threshold. Fig.12 In the illustrated example, the second row of the calibrated angle matrix 902 is the same as Fig.11 , 135]. In this regard, when the mobile device 150 compares the angle shown in row 2 of the calibrated angle matrix 902 to a predetermined threshold, the mobile device 150 generates a value of "1", as shown in row 3, column 2 and row 2, column 3 in the block matrix 1100. Referring to the block matrix 1100, all zero values do not indicate an outlier, and as noted above, an angle of -360 degrees corresponds only to a null value and can be ignored. In this case of the block matrix 1100, the third speaker 102c is determined to be an outlier relative to the second speaker 102b. The estimates in row 2, column 3 and row 3, column 2 will not be used in operations 606, 608, 610.
[0090] Fig.13 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 600 . The mobile device 150 can then check the block matrix 1100 to determine if there are any rows / columns filled with "1". As noted above, these values generally indicate that the speaker is an outlier. Fig.13 The illustrated block matrix 1100 is similar to the one shown in FIG. Fig.11The illustrated blocking matrix 1100 does not indicate the presence of any outliers. If the mobile device 150 does not detect an outlier in the blocking matrix 1100, the mobile device 150 generates an error and repeats the method 600 again.
[0091] Fig.14 and Fig.15 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 An example of microphone orientation and speaker layout estimation operation 608 of method 600. Fig.14 , the configuration of the first, second, third, and fourth speakers 102a to 102d as illustrated is now reflected as including distance coordinates in the x-axis and the y-axis. As shown, the first speaker 102a is selected as the reference speaker. The second speaker 102b has coordinates (100, 0) relative to the first speaker 102a, the third speaker 102c has coordinates (70.71, -70.71) relative to the first speaker 102a, and the fourth speaker 102d has coordinates (0, -100) relative to the first speaker 102a. As noted above, the mobile device 150 is unaware of the exact layout of the speakers 102a to 102d in the listening environment 151. In this way, the mobile device 150 establishes a reference matrix 1400 having reference coordinates (or distances or values): 0, 100, 100, 100, for the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. The mobile device 150 randomly selects coordinates (e.g., 0, 100, 100, 100). In this case, the mobile device 150 assumes that the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are positioned equidistantly from the first speaker 102a. As shown in the first matrix 800, the mobile device 150 has information corresponding to the angles of the audio received by the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d. However, the actual distances of such speakers 102a-102d are not known.
[0092] Reference now Fig.15 , the mobile device 150 calculates the distances (or x-coordinates, y-coordinates) of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a, which can be determined based on the following equation 9:
[0093] (x,y)(x s1 +Distance s1sj *cos(DOA s1sj ), y s1 -distance s1sj *sin(DOAs1sj )
[0094] For example, the mobile device 150 may calculate the distance coordinates of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a based on Equations 10, 11, and 12, respectively:
[0095] (x 2 ,y 2 )=(100*cos(0),-100*sin(0))=(100,0)
[0096] (x 3 ,y 3 )=(100*cos(45),-100*sin(45))=(70.71,-70.71)
[0097] (x 4 ,y 4 )=(100*cos(90), 100*sin(90))=(0,-100)
[0098] Equation 10 as shown above corresponds to the distance coordinate of the second speaker 102b relative to the first speaker 102a, where the angle of 0 is inserted into Equation 5 and is taken from the first row (i.e., S1) and the second column (i.e., S2) from the first matrix 800. Equation 11 as shown above corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle of 45 is inserted into Equation 8 and is taken from the first row (i.e., S1) and the second column (i.e., S3) from the first matrix 800. Equation 11 as shown above corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle of 90 is inserted into Equation 12 and is taken from the first row (i.e., S1) and the third column (i.e., S3) from the first matrix 800.
[0099] Figures 16 to 20 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 6. An example of a candidate coordinate estimation operation 610 of method 600. Fig.16It is generally illustrated that based on the performance of operation 608, the estimate of the layout of the third speaker 102c is positioned at coordinates (70.71, -70.71) relative to the first speaker 102a. However, the actual layout indicates that the third speaker 102c is positioned at coordinates (70.71, -70.71) relative to the first speaker 102a, the third speaker 102c is positioned at coordinates (100, -100) relative to the second speaker 102b, and the third speaker 102b is positioned at coordinates (-100, -100) relative to the fourth speaker 102d. These aspects are generally illustrated as candidate coordinate estimates 1600. Fig.17 An example is given based on the combination of Fig.15 Equations 10, 11, and 12 discussed above determine the manner in which the various coordinates of the third speaker 102c relative to the first, second, and fourth speakers 102a to 102d are determined. In general, the mobile device 150 does not know whether the coordinates of the third speaker 102c are correct. In this case, the mobile device 150 estimates possible candidate points. In operation 612, the mobile device 150 calculates the error for each candidate point. The candidate exhibiting the lowest error is selected as the best coordinate. It should be recognized that all DOA estimates from all speakers 102 are transmitted to the mobile device 150 using any number of wireless communication protocols (such as, but not limited to, Bluetooth, WiFi, etc.). Fig.17 In the illustrated example, the mobile device 150 combines the use of angles from the calibrated angle matrix 902 with determining the coordinates of the third speaker 102c relative to the first speaker 102a, the second speaker 102b, and the fourth speaker 102d.
[0100] refer to Fig.18 , the mobile device 150 expands the candidate coordinate estimate 1600 by combining the x-point and the y-point. As generally shown at 1800, the candidate coordinate estimate 1600 is provided in addition to the expanded candidate coordinate estimate 1802. In general, the candidate coordinate estimate reflects the x-coordinate and the y-coordinate in the following manner: (x a ,y a )、(x b ,y b ) and (x c ,y c ), respectively for the first speaker 102a, the second speaker 102b and the fourth speaker 102d. The expanded candidate coordinate estimate 1802 reflects the x-coordinate and the y-coordinate in the following manner: (x a ,y b )、(x c ,y z), respectively for the first speaker 102a and the second speaker 102b and the fourth speaker 102d and the first speaker 102a. The expanded candidate coordinate estimate 1802 is expanded in the manner illustrated at 1800 because a certain degree of estimation provides information for, for example, one dimension (e.g., x coordinate, y coordinate). The mobile device 150 combines the coordinates to obtain information in a two-dimensional format (e.g., x coordinate and y coordinate). The mobile device 150 typically aggregates the candidate coordinate estimate 1600, the expanded candidate coordinate estimate 1802, and the expanded angle as shown at 1804. The expanded angle 1804 is typically estimated using the angle corresponding to (100, -100) of the first speaker 102a and the second speaker 102b (e.g., S2 and S1). The first speaker 102a and the third speaker 102c form a line, and the coordinate (100, -100) is calculated using the intersection of the two lines. The angle information is used to form the line. The mobile device 150 expands the candidate coordinate estimates to locate the intersection of the third speaker 102c to the second speaker 102b and to the fourth speaker 102d. The mobile device 150 does not consider the third speaker 102c for the candidate coordinate estimates 1600, the expanded candidate coordinate estimates 1802, and the expanded angles 1804 because the location of the third speaker 102c is incorrect. In general, it is not necessary for the mobile device 150 to determine whether any location is correct. The numbers as set forth herein are provided merely as examples. Fig.18 The operation of the third speaker 102 c is disclosed as an example. In the whole method, these operations are applicable to each speaker 102 in the system 100 .
[0101] refer to Fig.19 , the mobile device 150 continues to expand the candidate coordinate estimate 1600. In this case, providing a combination of Fig.19 The example illustrated is only to illustrate another example of the expanded candidate coordinates, and combined with Fig.19 The illustrated examples may not be related to the examples shown above.
[0102] Fig. 20 Another example of predicting candidate coordinate estimation operations 610 by the mobile device 150 is depicted. Fig. 20Another example of the modified first matrix 800' and the modified blocking matrix 1100' is illustrated. The modified first matrix 800' indicates that there is an obstacle between the fourth speaker 102d and the third speaker 102c, as shown by the angle "145" in the third column (e.g., S3) and the fourth row (e.g., S4). In general, for a four-speaker layout configuration, the method 600 may only tolerate a single outlier between any two speakers. In this case, the mobile device 150 determines two or more outliers (e.g., the third speaker 102c and the fourth speaker 102d). The mobile device 150 determines that the third speaker 102c is an outlier, as discussed above in conjunction with operation 604. In this regard, the mobile device 150 also determines that the fourth speaker 102d is an outlier based on the description provided above in conjunction with operation 604. In view of the presence of more than one outlier, the mobile device 150 includes a user interface and commands the user to move any obstacles that exist relative to the third speaker 102c and the fourth speaker 102d. Given that the third speaker 102c and the fourth speaker 102d are outliers (i.e., have an obstacle formed between them), the mobile device 150 does not consider estimates from the third speaker 102c and the fourth speaker 102d for the candidate coordinate estimate 1600 (or vice versa), the expanded candidate coordinate estimate 1802, and the expanded angle 1804 because there is an outlier between the positions of the third speaker 102c and the fourth speaker 102d, and such estimates are considered incorrect. The mobile device 150 updates the modified block matrix 1100', which indicates that the third speaker 102c and the fourth speaker 102d are blocked in the layout for consideration.
[0103] Fig.21 and Fig. 22 Depicted is a pair according to one embodiment Figure 7 Configuration execution Figure 6 The mobile device 150 performs the optimal coordinate selection operation 612. For example, the mobile device 150 performs the following equation (13):
[0104]
[0105] in is the angle calculated by using the candidate x-coordinate and y-coordinate of the i-th speaker, and C corresponds to the index of the candidate. The mobile device 150 selects the candidate point that minimizes the error. The calibrated DOA matrix 800 as explained above is used as the DOA matrix in the above equation. ij .
[0106] Fig. 22 Examples of the manner of implementing Equation 13 are generally illustrated. It should be appreciated that if Fig. 22shown It can be obtained by the following equation:
[0107]
[0108] The mobile device 150 determines the errors of the third speaker 102c and the first speaker 102a respectively based on Equation 13:
[0109] Error 31 = |45-45| + |90-150| + |0-20| = 80
[0110] Similarly, the mobile device 150 also determines the errors of the third speaker 102c and the second speaker 102b respectively based on Equation 10:
[0111] Error 32 = |45-45| + |0-0| + |180-180| = 0
[0112] Fig. 22 This illustrates that equation 10 can be used for Fig. 22 The first two rows in the table in are used in a manner to make the third loudspeaker position estimation. comes from the second term in Equation 5. Fig. 22 As shown in the table represents the angle between the speaker 102 and the candidate point.
[0113] Fig.23 An example of a microphone orientation and speaker estimation method 600 is depicted according to one embodiment. Fig.23 The positions of the speakers 102a to 102d in the listening room 151 (or listening environment) are depicted. The mobile device 150 can display the positions (e.g., front, left, right, and rear) of the speakers 102a to 102d as arranged in the listening room. In general, the system 100 and / or method 600 is based on at least in conjunction with Figures 6 to 22 The method shown determines the positions of the first speaker, the second speaker, the third speaker, and the fourth speaker 102a to 102d in the listening environment 151. The system 100 and / or the method 600 utilizes the position information to provide channel allocation for immersive sound generation for the speakers 102a to 102d. For example, the mobile device 150 utilizes the final DOA to allocate the speakers 102a to 102d as front speaker positions, left speaker positions, right speaker positions, and rear speaker positions.
[0114] System and method for boundary distance estimation
[0115] As shown above, the first, second, third, and fourth speakers 102a-102d are typically from a family of products, all of which are equipped with microphones 106a-106b mounted thereon. The microphones 106a-106b of each speaker 102 provide the ability to detect the location of an audio source (e.g., a mobile device 150) relative to any nearby walls. However, since the microphones 106a-106b may be arranged linearly when packaged on the corresponding speaker 102, the microphones 106a-106b may lack the ability to distinguish between audio sources located in front of or behind the speaker based on using the line between the microphones 106a-106b as a line of symmetry. Detecting a wall or obstacle in one of the directions can eliminate the symmetry limitation.
[0116] In addition, if the speaker is placed too close to a wall or a corner, the speaker may not be detected. The disclosed system can detect whether the speaker is placed too close to the wall, and automatically correct the speaker being positioned too close to the wall to ensure that the desired sound field is transmitted in the room (or listening environment 151). In general, a speaker closer to the wall may have a + / -3dB effect at low frequencies. In addition, the disclosed system and method can be used for improved audio upmixing. The various aspects disclosed herein can provide, for example, a circular microphone array with six microphones, which can detect all surrounding walls using the disclosed method. At this point, the disclosed method can determine whether the left wall or the right wall is the surrounding wall of the microphone by comparing the proximity to the wall of each microphone. At this point, the system can perform channel allocation, which can be used to perform upmixing that can be performed automatically. In addition, the disclosed system and method can obtain room characteristics and estimate the distance from the wall or reflector.
[0117] The room impulse response (RIR) generally provides an audio fingerprint of a location in an acoustic environment. There may be various RIR applications, such as wall boundary estimation, digital reconstruction of acoustic environments for professional audio applications, room correction, and frequency response correction for playback systems. The measurement of the RIR includes exciting the room (or listening environment), which can be performed by (but not limited to) clapping. The measurement of the RIR may also include deconvolving the audio signal to obtain room characteristics. The RIR may involve reflections after the room is excited. Reverberation may refer to audio reflections reflected back to the audio source. Reverberation is usually not a direct sound, so the reverberation arrives at the microphone later. The reverberation amplitude and return time depend on the material of the surface and the amount of reflected area. The sound continues to reflect until the sound loses its energy due to absorption.
[0118] Fig.24A system 2400 for performing boundary estimation according to one embodiment is depicted. The system 2400 generally includes a first speaker 102a and a second speaker 102b. Each of the speakers 102a, 102b generally includes an audio source 2402. Although only the first speaker 102a and the second speaker 102b are shown, it should be appreciated that any number of speakers may be positioned in the listening environment. The audio source 2402 may be integrated into any one of the speakers 102a, 102b to transmit audio from a particular speaker 102 directly into the listening environment 151.
[0119] The first speaker 102a and the second speaker 102b are located at a certain distance from the wall 2404. In general, it is desirable to know the distance of the first speaker and / or the second speaker 102a-102b from the wall 2404 in the listening environment 151. If one or more of the first speaker and the second speaker 102a-102b are placed too close to the wall 2404, this condition may make it difficult for the audio source 2402 to automatically correct the position of the wall 204 relative to the speakers 102a-102b to ensure that the desired sound field is transmitted into the room (or listening environment 151). In general, the first speaker and / or the second speaker 102a may cause a + / - 3dB effect at low frequencies when they are respectively positioned too close to the wall 2404. The audio source 2402 (i.e., within the speaker 102a and / or the speaker 102b) can determine the position of the first speaker and / or the second speaker 102a-102b relative to the wall 2404 and employ a correction mechanism to account for the distance of the first speaker and / or the second speaker 102a-102b that is positioned too close to the wall 2404. The system 2400 can improve channel assignments using more than two microphones 106a by employing a correction mechanism to account for the close proximity of the speakers 102a-102b to the wall 2404. The ability to correctly perform channel assignments (e.g., which speaker is the left front / right front / rear speaker, etc.) enables audio upmixing. It should be appreciated that the audio source 2402 can include any number of controllers 2410 (hereinafter referred to as "controllers 2410") to perform the operations noted herein. Although the audio source 2402 can determine the distance of the first speaker and / or the second speaker 102a to 102b relative to the wall 2404, it should be recognized that any one or more of the first speaker 102a or the second speaker 102b can also include at least one controller 2412 to determine the distance of the speakers 102a, 102b relative to the wall 2404.
[0120] The controller 2410 may provide and transmit the audio fingerprint in the listening environment 151 using, for example, a predetermined measurement scheme such as RIR. For example, the controller 2410 may include a driver (not shown) to transmit the audio fingerprint into the listening environment 151. The controller 2410 may also include a memory to store the audio fingerprint. The system 2400 may employ various RIR applications, such as wall boundary estimation, digital reconstruction of the acoustic environment for professional audio applications, room correction, and frequency response correction for playback systems. In one example, the audio source 2402 may excite the room (or listening environment 151) by transmitting an audio signal and perform, and the measurement of the RIR may also include deconvolution of the audio signal to obtain room characteristics. As noted above, the RIR may involve performing a measurement of the captured audio fingerprint (i.e., reflection) after the listening room 151 has been excited. Reverberation may refer to audio reflections reflected back to the audio source 2402. The audio source 2402 may be coupled to microphones 106a and 106b to receive the captured reflections (or reverberation) from the listening environment 151. Reverberation, as received back by the audio source 2402, is typically not a direct sound, so the reverberation arrives at the microphone 106 at a later time. The amplitude of the reverberation and the time it takes for the reverberation to arrive at the audio source 2402 depends on the material of the surface within the listening environment 151 and the amount of reflective area. Sound continues to reflect until the sound loses its energy due to absorption within the listening environment 151.
[0121] The audio source 2204 may excite the listening environment 151 by transmitting an audio signal comprising an exponential sine sweep (ESS) (or an ESS signal). The ESS signal may have advantages over impulse response measurement methods because (i) for signals transmitted with the same length as the length of a maximum length sequence (MLS), the ESS signal has better noise suppression than the MLS, and (ii) given that the driver transmits the ESS signal directly, the ESS signal may be more robust to nonlinear effects.
[0122] For the ESS signal, the following equation can be provided:
[0123]
[0124] in:
[0125]
[0126] T represents the duration of the scan. The variable ω 1 and ω 2 Corresponding to the starting frequency and the ending frequency respectively. Since the frequency of ESS varies, the energy may depend on the rate of instantaneous frequency given by:
[0127]
[0128] Fig.25 Illustrated generally is a frequency response of an ESS signal 2500. When the signal 2500 is transmitted from the audio source 2402 to one or more of the first speaker 102a and the second speaker 102b, the ESS signal 2500 includes a peak 2502 thereof.
[0129] The audio source 2402 may employ inverse filtering or deconvolution to measure the RIR after the first speaker 102a and / or the second speaker 102b plays the EES signal 2500 in the listening environment 151. The controller 2410 then employs inverse filtering and extracts the RIR. As noted above, the audio source 2402 includes any number of microphones 2420 to record the ESS signal 2500. The audio source 2402 may then extract or measure the RIR from the recorded ESS signal 2500. The time-reversed energy of the ESS signal 2500 is reduced, for example, at 3 dB / octave, and the inverse filter has, for example, a 3 dB / octave increase in its energy spectrum to achieve a flat spectrogram. Assume that h(t) is the room impulse response, r(t) is the excited room impulse response, and f(t) is the inverse filter.
[0130] h(t)=r(t)*f(t) (13)
[0131] f(t) can be created using post-modulation, which is to apply an amplitude modulation envelope of +6dB / octave to the spectrum of the time-reversed signal. The general form of the post-modulation function is as follows:
[0132]
[0133] A represents the constant of the modulation function. For time t = 0, ω(t) = w 1 , and for unity gain at time t = 0:
[0134]
[0135] Then, the modulation function becomes:
[0136]
[0137] f(t) has a 3dB / octave increase in frequency after modulating the time-reversed signal with m(t). Fig.26 The amplitude spectrum of the inverse filter is illustrated.
[0138] In general, the measured RIR is provided by the audio source 2402 using Equation 13. Thus, aspects associated with Equation 13 correspond to the convolution of the ESS signal and the inverse filter. The audio source 2402 may use the measured RIR to estimate the distance of the first speaker 102a and / or the second speaker 102b to the wall 2404. It should be appreciated that the audio source 2402 given the speakers 102a and 102b determines the distance of each speaker 102a and 102b at which the audio source 2402 is located. In general, the wall proximity estimate as used by the audio source 2402 may be complex because the measured RIR includes reverberation from multiple objects in the listening environment 151 (or room).
[0139] Fig. 27 An example of a graph 2700 corresponding to an RIR measurement as performed by an audio source 2402 is generally illustrated. As shown, the RIR measurement 2700 includes a plurality of peaks 2702. The peaks 2702 may correspond to reflections or reverberations of the ESS signal from various objects in the listening environment 151. A reverberation number 1850 is generally shown at 2704. The reverberation number 1850 generally corresponds to reverberation of the ESS signal from the wall 2404. This condition may be verified because the amplitude of the peak is highest after a gap 2710 is shown between the highest peak 2712 and the reverberation number 1850, as shown at 2704. The highest peak 2172 generally indicates a direct path of the ESS signal from the speaker 102 to the microphone 2420. Additionally, since the peak amplitude associated with 2704 may correspond to the material of the wall 2404, this secondary condition may also be verified. Generally speaking, the peak from the wall 2404 may not be obvious, as shown at 2704. Fig. 27 In addition, the nonlinearity attributed to peak 2702 may be caused by a driver (or amplifier) in audio source 2402. For example, an amplifier often causes a false peak in the RIR measurement performed by audio source 2402. Therefore, audio source 2402 may need to account for these conditions when performing RIR measurements.
[0140] The audio source 2402 can overcome the above problems to perform wall distance estimation by: (i) sampling or extracting the peaks in the RIR measurement to avoid strong pseudo peaks (or ringing) close to the peak to be detected around the peak 2702, which may cause erroneous estimates; and / or (ii) scoring each peak to determine the correct peak from the wall 2404. It should be recognized that due to nonlinearity, there are undesirable peaks around the peak 2702, and it is desirable to avoid such peaks in the RIR measurement. In general, the peak 2702 in the RIR measurement can correspond to the direct path from the audio source 2402 to the microphone 2420 and from the reflector to the microphone 2420 on the audio source 2402. A careful observation of the RIR measurement can notice the presence of ringing around the peak. The audio source 2402 can extract the peak to detect the impulse event. Therefore, in this regard, the audio source 2402 can use a sliding window to extract the peak in each window. The audio source 2402 may find each peak in the window after obtaining the largest peak in the RIR measurement and ignore other peaks in the RIR measurement.
[0141] Fig.28 An RIR measurement 2800 is generally illustrated with a peak 2802 detected by an audio source 2402 according to one embodiment. Fig.28 Also exemplified as above combined Fig. 27 The noted ringing is more pronounced or noticeable. When the distance from the first speaker 102a and / or the second speaker 102b to the wall is 137 cm (for a sampling rate of 32 kHz), the audio source 2402 obtains the RIR measurement 2800. The distance from the speaker to the wall can be obtained based on the following equation:
[0142]
[0143] For example, the "index of the estimated peak value" set forth above in Equation 17 generally corresponds to the estimated peak in RIR measurement 2800. Thus, in this regard, Fig.28 The detected peak 2802a shown corresponds to sample 251, which can be defined as the index of the estimated peak. The "index of the largest peak" can generally correspond to zero. Fig.28 Generally describes Fig. 27 A pruned version of the RIR measurements shown. Fig. 27 The highest peak 2712 shown generally corresponds to Fig.282802 shown at 0 on the x-axis of the graph of . Similarly, the constant value "2" as set forth in Equation 17 generally corresponds to the round trip time. When the audio source 2402 plays the ESS signal, the sound (or ESS signal) contacts the wall 2404 and reflects back to the audio source 2402. Therefore, the ESS signal takes the same distance or path twice. Therefore, in this regard, it is explained why the constant value is set to 2. For example, Fig.28 28. The illustrated RIR measurement 2800, if the audio source 2402 correctly estimates the peak at sample 251, then the audio source 2402 determines the distance between the wall 2404 and the first speaker 102a or the second speaker 102b using equation 17 from above as, for example, 137 cm for a 32 kHz sampling rate.
[0144] The audio source 2402 tracks the overall trend of the peaks 2802 of the RIR measurement 2800 to estimate the peaks of the reverberation of the RIR measurement 2800. For example, if the ESS signal as transmitted by the audio source 2402 does not encounter a wall 2404 or object in the listening environment 151, the expected trend of the peaks 2802 of the RIR measurement will indicate or correspond to an overall decrease in the peaks (i.e., a downward trend). If the ESS signal as transmitted by the audio source 2402 does encounter a wall 2404 or object in the listening environment 151, the expected trend of the peaks 2802 of the RIR measurement will indicate a downward trend in the peaks 2802, followed by an upward trend in the peaks, and then followed by a downward trend in the peaks 2802. In general, the audio source 2402 stores information corresponding to the peaks 2802 as received for the RIR measurement to determine whether there is only a downward trend in which the peaks 2802 continue to decrease over time, or whether there is a downward trend in the peaks 2802, followed by an upward peak 2802a. The audio source 2402 can then establish a confidence score, which is calculated by using, for example, the percentage of rise multiplied by a value of, for example, 1.01, the number of negative peaks 2802. The audio source 2402 can then select a predetermined number of peaks with the highest confidence score (i.e., the maximum score) or level (e.g., 20), and then locate the largest peak among the selected peaks 2802. Such a largest peak can correspond to a peak that exhibits the largest amplitude on the RIR measurement, and may be positive after a long series of descending peaks. In this case, the largest peak can be selected as the sample number (e.g., 251), which is then utilized by the audio source 2402 for insertion into Equation 17, as provided above, to find the distance of the speaker 102a or 102b from the wall 2404.
[0145] Fig.29A method 2900 for performing boundary estimation involving multiple speakers 102 is depicted according to one embodiment. In operation 2902, an audio source 2402 transmits an audio signal in the form of an ESS signal into a listening environment 151. It should be appreciated that the audio source 2402 may transmit one ESS signal at a time from each speaker 102 positioned in the listening environment 151 and perform the operations of the method 2900 for each speaker 102 to determine the distance of the speaker 102 relative to a wall 2404. Each audio source 2402 determines the distance of its corresponding speaker 102a, 102b relative to the wall 2404. It should be appreciated that each speaker 102a, 102b may also transmit the distance information to a mobile device 150 or other device that may require such information so that the mobile device or other audio source can compensate the audio output to mitigate the situation where one or more of the speakers 102a, 102b are too close to the wall 2404.
[0146] In operation 2904, the audio source 2402 receives reverberation from the listening environment 151 in response to transmitting the ESS signal. In this case, the audio source 2402 detects the peak 2802 of the reverberation in the RIR measurement 2800 and stores in its memory information corresponding to the peak 2802. In operation 2906, the audio source 2402 performs trend tracking on the peak 2802.
[0147] In operation 2908, the audio source 2402 evaluates the stored peaks 2802 of the reverberation to determine whether there is only a decreasing trend in the RIR measurement with the peak 2802 decreasing over time, or whether there is a decreasing trend in the RIR measurement with the peak 2802 followed by an increasing peak 2802a. If the audio source 2402 determines that the peak 2802 is not increasing over time, the method 2900 moves to operation 2912 and determines that the wall distance of the first speaker 102a or the second speaker 102b cannot be determined. In this case, the method 2900 can return to operation 2902. If the audio source 2402 determines that there is an increasing peak 2802a in the RIR measurement, the method 2900 moves to operation 2910.
[0148] In operation 2910, the audio source 2402 establishes a confidence score, which is calculated by using, for example, the percentage of rise multiplied by a value such as 1.01 for the number of negative peaks 2802. The audio source 2402 can then select a predetermined number of peaks with the highest confidence score or level (e.g., 20), and then locate the largest peak among the selected peaks 2802. Such a largest peak can correspond to the peak 2802a that exhibits the largest amplitude on the RIR measurement, and may be positive after a long series of descending peaks 2802. In operation 2912, the audio source 2402 applies the largest peak to a distance equation (e.g., Equation 17), and also applies other variables as noted above in conjunction with Equation 17 to determine the distance of the first speaker 102a or the second speaker 102b relative to the wall 2404.
[0149] It should be appreciated that the controller as disclosed herein may include various microprocessors, integrated circuits, memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variants thereof) and software that cooperate with each other to perform the (multiple) operations disclosed herein. In addition, such controllers as disclosed utilize one or more microprocessors to execute a computer program embodied in a non-transitory computer-readable medium, the computer program being programmed to perform any number of functions as disclosed. In addition, the (multiple) controllers provided herein include a housing and various numbers of microprocessors, integrated circuits, and memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) positioned within the housing. The (multiple) controllers as disclosed also include hardware-based inputs and outputs, which are used to receive data from and transmit data to other hardware-based devices as discussed herein, respectively.
[0150] Although the above describes exemplary embodiments, it is not meant that these embodiments describe all possible forms of the present invention. On the contrary, the words used in this specification are descriptive words rather than limiting words, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. In addition, the features of various implementation embodiments can be combined to form other embodiments of the present invention.
Claims
1. An audio system, the audio system include: a speaker for transmitting an audio signal into a listening environment defined by at least one wall of the room; at least one microphone positioned on the speaker and configured to capture a reverberant audio signal comprising a plurality of reverberations and a plurality of peaks, wherein the reverberant audio signal indicates that the audio signal is reflected from the at least one wall; and at least one controller programmed to: applying a confidence score to the plurality of peaks to obtain a maximum score, the maximum score indicating a largest peak of the audio signal reflected from the at least one wall; and A distance between the speaker and the wall is determined based at least on the maximum score.
2. The audio system of claim 1, wherein the at least one controller is further programmed to execute a predetermined measurement scheme to obtain measured values of a plurality of peaks associated with the plurality of reverberations.
3. The audio system of claim 2, wherein the predetermined measurement scheme is a room impulse response (RIR) measurement.
4. The audio system of claim 2, wherein the at least one controller is further programmed to determine whether the plurality of peaks exhibit at least one of an increasing trend and a decreasing trend over a period of time.
5. The audio system of claim 4, wherein at least one controller is further programmed to determine the distance of the speaker relative to the at least one wall in response to the plurality of peaks exhibiting a decrease and then an increase over the period of time.
6. The audio system of claim 5, wherein the at least one controller is further programmed to select a predetermined number of the plurality of peaks that exhibit maximum amplitudes.
7. The audio system of claim 6, wherein the at least one controller is programmed to apply the confidence score to the plurality of peaks exhibiting the maximum amplitude to obtain the maximum score, the maximum score being indicative of the largest peak reflected from the at least one wall.
8. The audio system of claim 7, wherein the at least one controller is further programmed to determine the distance between the speaker and the wall based at least on the maximum peak and sound velocity.
9. The audio system of claim 1, wherein the audio signal is an exponential sine sweep signal.
10. The audio system of claim 1, wherein the at least one controller is positioned in the speaker.
11. A method, the method comprising: include: transmitting the audio signal via the loudspeaker into a listening environment defined by at least one wall in the room; capturing, via at least one microphone, a reverberant audio signal comprising a plurality of reverberations and a plurality of peaks, wherein the reverberant audio signal indicates that the audio signal is reflected from the at least one wall; applying a confidence score to the plurality of peaks to obtain a maximum score, the maximum score indicating a largest peak of the audio signal reflected from the at least one wall; as well as A distance between the speaker and the wall is determined based at least on the maximum score.
12. The method of claim 11, further comprising executing a predetermined measurement protocol to obtain measurement values of a plurality of peaks associated with the plurality of reverberations.
13. The method of claim 12, wherein the predetermined measurement scheme is a room impulse response (RIR) measurement.
14. The method of claim 12, further comprising determining whether the plurality of peaks exhibit at least one of an increasing trend and a decreasing trend over a period of time.
15. The method of claim 14, further comprising determining the distance of the speaker relative to the at least one wall in response to the plurality of peaks exhibiting a decrease and then an increase over the period of time.
16. The method of claim 15, further comprising selecting a predetermined number of the plurality of peaks that exhibit a maximum amplitude.
17. The method of claim 16, further comprising applying the confidence score to the plurality of peaks exhibiting the maximum amplitude to obtain the maximum score, the maximum score being indicative of the largest peak reflected from the at least one wall.
18. The method of claim 17, wherein determining the distance further comprises determining the distance between the speaker and the wall based at least on the maximum peak and velocity of sound.
19. The method of claim 11, wherein the audio signal is an exponential sine sweep signal.
20. A computer program product embodied in a non-transitory computer readable medium stored in a memory, the computer program product being programmed and executable by at least one controller in an audio system, the computer program product comprising instructions to: transmitting the audio signal via the loudspeaker into a listening environment defined by at least one vertical surface in the room; capturing a reverberant audio signal comprising a plurality of reverberations and a plurality of peaks, wherein the reverberant audio signal indicates that the audio signal is reflected from the at least one vertical surface; applying a confidence score to the plurality of peaks to obtain a maximum score, the maximum score indicating a largest peak of the audio signal reflected from the at least one vertical surface; and A distance between the speaker and the at least one vertical surface is determined based at least on the maximum score.
