Devices, systems and / or methods for loudspeaker audio calibration

Through the sound noise-rosting time-frequency masking technology and phase conversion technology, the problem of automatic calibration and arrival direction estimation of speakers in low signal-to-noise ratio environments is solved, and the accurate automatic calibration of speakers and immersive sound generation is realized, improving the sound quality and Robustness.

CN119071696APending Publication Date: 2024-12-03HARMAN INT IND INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410635281.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-05-22
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing speakers are difficult to achieve accurate automatic calibration and arrival direction estimation in low signal-to-noise environments, resulting in a degradation in immersive sound generation quality and inability to effectively handle background noise and reverb.

Method used

Using noise-rosive time-frequency masking technology, the signature sound signal is extracted through inverse exponential sinusoidal scanning signal and generalized cross-correlation phase transformation technology, time-frequency masking operation is performed to reduce the impact of noise, determine the arrival direction of the speaker, and in the mobile device Perform microphone orientation estimation and outlier detection to optimize speaker configuration.

Benefits of technology

Accurate automatic calibration and arrival direction estimation of speakers in low signal-to-noise ratio and reverberation environments, improve the quality of immersive sound generation, can handle corner and background noise, and provide robust speaker configuration and layout estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071696A_ABST
    Figure CN119071696A_ABST
Patent Text Reader

Abstract

An audio system including a first speaker and a second speaker is provided. The first speaker transmits an audio signal including audio data and a signature tone into a listening environment. The second speaker includes at least one controller programmed to receive the audio signal including the audio data and the signature tone, and determine a direction of arrival (DOA) of the audio signal as received at the second speaker based at least on the signature tone. The at least one controller is further programmed to perform a time-frequency masking operation on the received audio signal prior to determining the DOA of the audio signal.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is generally related to U.S. application No. 18 / 204,150, filed on May 31, 2023, entitled “SYSTEM AND / OR METHOD FOR LOUDSPEAKER AUTO CALIBRATION AND LOUDSPEAKER CONFIGURATION LAYOUT ESTIMATION” (Agent File No. P220102US (HARM0866PUS)), the disclosure of which is hereby incorporated by reference in its entirety.

[0003] This application is generally related to U.S. application No. 18 / 204,159, filed on May 31, 2023, entitled “BOUNDARY DISTANCE SYSTEM AND METHOD” (Agent Docket No. P220103US (HARM0868PUS)), the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0004] Various aspects disclosed herein generally relate to an apparatus, system and / or method for noise robust time-frequency masking based direction of arrival estimation for speaker audio calibration. These and other aspects will be discussed in more detail herein. Background Art

[0005] Various speaker manufacturers or providers may group various speaker categories together to form an ecosystem. In this regard, the various speakers communicate or work with each other and / or with the mobile device. Thus, such speakers may achieve higher audio quality using immersive sound. Information related to the position of the speakers may be required to generate immersive sound. Therefore, automatic calibration may be required before the speakers can generate immersive sound. Summary of the invention

[0006] In at least one embodiment, an audio system is provided that includes a first speaker and a second speaker. The first speaker transmits an audio signal that includes audio data and a signature tone to a listening environment. The second speaker includes at least one controller that is programmed to receive the audio signal that includes the audio data and the signature tone, and to determine a direction of arrival (DOA) of the audio signal as received at the second speaker based at least on the signature tone. The at least one controller is also programmed to perform a time-frequency masking operation on the received audio signal before determining the DOA of the audio signal.

[0007] In at least another embodiment, an audio system is provided that includes a first speaker. The first speaker includes a memory and at least one controller. The at least one controller is programmed to receive an audio signal including audio data and a signature tone from a second speaker, and store the audio signal including the signature tone in the memory. The at least one controller is also programmed to determine a direction of arrival (DOA) of the audio signal based at least on the signature tone after performing a time-frequency masking operation, and to perform a time-frequency masking operation on the received audio signal before determining the DOA of the audio signal.

[0008] In at least another embodiment, a computer program product embodied in a non-transitory computer readable medium is provided, the computer program product being stored in a memory and being programmed and executed by at least one controller in an audio system. The computer program product includes instructions for receiving an audio signal from a second speaker at a first speaker, the audio signal including audio data and a signature tone; and for determining a direction of arrival (DOA) of the audio signal as received at the first speaker based at least on the signature tone. The computer program product also includes instructions for performing a time-frequency masking operation on the received audio signal before determining the DOA of the audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present disclosure are particularly pointed out in the appended claims. However, other features of the various embodiments will become more apparent and are best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 Generally, a system for performing noise robust time-frequency masking based directional and speaker automatic calibration and speaker configuration layout estimation is shown according to one embodiment;

[0011] Figure 2 shows how to remove noise by applying time-frequency (TF) masking according to one embodiment;

[0012] Figure 3 shows the output of the TF masking block according to one embodiment;

[0013] Figure 4 Two-microphone direction of arrival estimation using time difference of arrival is shown;

[0014] Figure 5 Various signature tone signals are shown according to one embodiment;

[0015] Figure 6 A method for performing optimized speaker auto-calibration and speaker configuration estimation according to one embodiment is shown;

[0016] Figure 7 shows an example of a speaker and microphone configuration according to one embodiment;

[0017] Figures 8 to 9 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and speaker placement estimation operations of the method;

[0018] Figures 10 to 12 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Example of outlier detection operation of the method;

[0019] Fig.13 According to one embodiment, Figure 7 Executed on the configuration Figure 6 An example of a reference speaker selection operation of the method;

[0020] Fig.14 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and speaker placement estimation operations of the method;

[0021] Fig.15 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and speaker placement estimation operations of the method;

[0022] Fig.16 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the candidate coordinate estimation operation of the method;

[0023] Fig.17 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the candidate coordinate estimation operation of the method;

[0024] Fig.18 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the candidate coordinate estimation operation of the method;

[0025] Fig.19 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the candidate coordinate estimation operation of the method;

[0026] Fig. 20 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the candidate coordinate estimation operation of the method;

[0027] Fig.21 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Examples of the best coordinated selection operations of the method;

[0028] Fig. 22 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Another example of the best coordination selection operation of the method;

[0029] Fig.23 An example of microphone orientation and speaker estimation according to one embodiment is shown;

[0030] Fig.24 A system for performing boundary estimation according to one embodiment is shown;

[0031] Fig.25 shows an example of a frequency response of an exponential sine sweep (ESS) for stimulating a listening environment according to one embodiment;

[0032] Fig.26 shows an example of an amplitude spectrum of an inverse filter according to one embodiment;

[0033] Fig. 27 An example of a room impulse response (RIR) measurement according to one embodiment is shown;

[0034] Fig.28 shows an example of peak detection involving RIR measurements according to one embodiment; and

[0035] Fig.29 A method for performing boundary estimation involving multiple speakers according to one embodiment is shown. DETAILED DESCRIPTION

[0036] As required, detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to employ the present invention in various ways.

[0037] Noise-robust time-frequency masking based direction-of-arrival estimation for loudspeaker automatic calibration

[0038] The speaker is used to generate an immersive sound effect. For one aspect of immersive sound, it is necessary to perform automatic calibration to locate the position of the speaker. A method for performing speaker positioning includes estimating the azimuth of the speaker, which is also called the direction of arrival estimation (DOA). The performance of the DOA method may be problematic for low signal-to-noise ratios (SNR) (i.e., less than 0 dB) because noise is the main signal under low SNR conditions. In addition, in real-world scenarios, noise may not be avoided for the automatic calibration stage. Therefore, the automatic calibration stage requires a noise-robust DOA estimation method. The disclosed system and / or method utilizes time-frequency (TF) masking that can be used for source separation as a preprocessing step of the DOA estimation method to achieve high performance at low SNR. TF masking can extract the desired signal from the noise signal captured by the microphone. The various aspects provided herein also provide a signature signal that maximizes performance under low SNR conditions. The embodiments disclosed herein provide a DOA estimation based on TF masking designed using at least two microphones and a signature signal that can be played back by a speaker.

[0039] As described above, for immersive sound generation of speakers, automatic calibration is generally required. Failure in the automatic calibration phase may lead to negative feedback from the listener. In addition, since the environment cannot be controlled during the automatic calibration phase, background noise cannot be avoided. Therefore, for immersive sound generation using multiple speakers, noise robust automatic calibration is desired. The disclosed embodiment provides noise robust automatic calibration to provide immersive sound generation. In addition, the disclosed system generally provides accurate DOA estimation for speaker automatic calibration under low signal-to-noise ratio and reverberation. These aspects can realize immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner situations (such as two speakers, for example, on the same line but not aligned with each other).

[0040] One approach to speaker auto-calibration may involve estimating the angle of the speaker, also referred to as DOA. There are many techniques for estimating the DOA of a talker / speaker, such as time difference of arrival (TDOA), multiple signal classification (MUSIC), and steerable response power (SRP). While the TDOA approach does not provide satisfactory performance at low signal-to-noise ratios (SNRs), MUSIC and SRP require a large number of microphones to achieve high performance at low SNRs. Even the performance of the MUSIC and SRP approaches falls short of the requirements for low SNR conditions (i.e., -10 dB babble noise). The disclosed system provides a signature tone in the form of an inverse exponential sine sweep (ESS) signal, which has been found to provide an indication to a controller, inter alia, to initiate speaker auto-calibration in noisy environments, such as -10 dB. In these types of noisy environments, the controller may not be able to perceive other types of signature tones that do not utilize ESS-based signals.

[0041] Figure 1 A system 100 for performing noise robust time-frequency masking based directional and speaker automatic calibration and speaker configuration layout estimation according to one embodiment is shown. The system 100 includes a speaker 102 having a plurality of microphones 106a to 106b (or "106"), a time-frequency (TF) masking block 108, a signature frame detection block 110, a generalized cross-correlation (GCC) phase transform (PHAT) block 112, at least one controller 122 ("controller 122"), and a memory 130. The system 100 also includes a mobile device 150 having a matrix array 114, a microphone orientation estimation (MOE) block 116, an outlier detection block 118, an optimization block 120, and at least one controller 123 (hereinafter referred to as "controller 123"). It should be appreciated that the controller 123 may execute any instructions for any function performed by the mobile device 150 as described herein. Although Figure 1 A single speaker 102 is shown, but it should be appreciated that the system 100 includes any number of speakers 102 positioned therein.

[0042] At least one of the speakers 102 transmits an audio signal including the signature sound 104 into the listening environment 151 to the other speakers 102 in the system 100. It should be appreciated that the speakers 102 typically include at least two of the microphones 106a-106b. The speakers 102 may transmit an audio signal including the signature sound 104 into the listening environment 151. The microphones 106a-106b positioned on different speakers 102 capture the audio signal including the signature sound 104. Each speaker 102a and 102b includes a memory 130. The memory 130 of the speaker 102b stores the audio signal and the corresponding signature sound (or signature frame) 104 for processing.

[0043] As described above, the TF masking block 108, the signature frame detection 110, the GCC PHAT block 112, and the controller 122 are implemented in all speakers 102 present in the system 100. For example, assume that the system 100 includes four speakers 102, and the first speaker 102 receives audio signals and corresponding signature tones 104 from the other speakers 102. Therefore, in this regard, each speaker 102 estimates the direction of arrival (DOA) of the audio signals received from the other three speakers 102. The mobile device 150 includes one or more transceivers 155 to wirelessly receive the DOA estimates from each of the speakers 102 in the system 100. It is also recognized that each of the speakers 102 in the system 100 may also include one or more transceivers 152 to wirelessly transmit the estimated DOA information to the mobile device 150.

[0044] In general, the TF masking block 108 in the speaker 102 can reduce the noise effect associated with the captured audio signal received from other speakers 102 in the system 100. For example, the controller 122 applies the TF masking block 108 to each microphone input to reduce the noise effect. After the TF masking block 108 reduces the noise effect, the signature frame detection block 110 estimates the signature tone 104. In one example, the length of the signature tone 104 can be 200 milliseconds. However, the speaker 102 records the received audio, for example, more than 200 milliseconds, because the speaker 102 does not know when another speaker 102 plays the signature tone 104. It can be assumed that the speaker 102 may be in a recording mode while another speaker 102 transmits the signature tone 104. It is generally desirable to detect the signature tone 104 for a sufficiently long duration to correctly estimate the DOA. Receiving the signature tone 104 on the audio signal can indicate to the receiving speaker 102 that the system 100 may be in an automatic calibration mode. In the automatic calibration mode, the speakers 102 may transmit information corresponding to the positions of the speakers 102 relative to the mobile device 150 (or other audio source).

[0045] The controller 122 cross-correlates the signature tone 104 played by the transmitting speaker 102 with the acquired audio. The cross-correlation performed by the GCCPHAT block 112 can provide the location of the signature tone 104 in the long recording. In this regard, the controller 122 uses the location to extract the signature tone 104. At this point, the extracted signature tone 104 is provided to the GCC-PHAT block 112. The controller 122 can then use the estimated DOA to perform automatic calibration of the speaker 102b. These aspects will be discussed in more detail below. Returning to the TF masking block 108, the controller 122 applies a TF masking operation as a preprocessing step for DOA estimation. The TF masking block 108 can eliminate the TF frequency band with the most noise in the audio signal to minimize the effects of noise and reverberation. The noisy input audio signal including the signature tone 104 is combined with Figure 2 As shown generally at 200. Figure 2 The noisy input audio signal including the signature tone 104 shown comprises a noisy swept sine between 6 kHz and 7 kHz. Figure 2 202 shows generally that, in response to the TF masking module 108 performing a TF masking operation, the controller 122 extracts the signature tone 104 or signal from the noise mixture of the input audio signal.

[0046] Return to reference Figure 1 , TF masking block 108 uses techniques for source separation and speech enhancement. TF masking block 108 receives signature tone 104 to generate an enhanced signal. Controller 122 generates sampling delays using signature tone 104 via the enhanced signal The controller 122 uses the sampling delay To determine the DOA of the audio signal received at the receiving speaker 102. The TF masking-based techniques described above may include ideal binary masking (IBM), ideal ratio masking (IRM), complex ideal ratio masking (cIRM), optimal ratio masking (ORM), etc. In general, when TF-based DOA estimation is adopted, such masking techniques should not modify the phase information. Based on various requirements and tests, the disclosed system 100 may adopt an IRM defined by:

[0047]

[0048] References to Equation 1 may be found, for example, in "The Optimal Ratio Time-Frequency Mask for Speech Separation in Terms of Signal-to-Noise Ratio," The Journal of the Acoustical Society of America 134, No. 5 (2013): EL452-EL458. When S(t, f) is the frequency response of the signature signal (or signature tone 104), N(t, f) represents the noise spectrum and β is a smoothing factor. Due to the overall knowledge of the signature tone 104, S(t, f) can be calculated. The denominator in equation (1) can be the captured signal at microphones 106a to 106b. After the controller 122 calculates the mask, the product of the captured signal and the mask can be used to calculate the enhanced signal, as shown in equation (2).

[0049] E(t,f)=IRM(t,f)*Y(t,f) (2)

[0050] E(t, f) represents an enhanced signal, which is a two-channel signal given that the two microphones 106a and 106b of the receiving speaker 102 each receive an input audio signal including the signature tone 104. Y(t, f) corresponds to the captured signal at the microphones 106a to 106b. The enhanced signal may correspond to Figure 2 0045] FIG . 1 is a signal generally shown at 202 in which noise is removed from a captured audio signal.

[0051] Figure 3 An example of a recorded signal 300 provided by the TF masking block 108 of the speaker 102 according to one embodiment is shown. For example, the recorded signal 300 corresponds to the output provided by the TF masking block 108 after the TF masking block 108 performs a masking operation. In general, the recorded signal 300 provided by the TF masking block 108 may include a long string of audio data 302, which includes silence / noise and signature sounds 104. The signature sounds 104 are generally delimited by frames 304. As shown in FIG. Figure 3 The two audio signals shown correspond to one audio signal received at microphone 106a and another audio signal received at microphone 106b, which are then processed by TF masking block 108. Controller 122 detects frame 304 using cross-correlation between audio data 302 and signature tone 104. In general, controller 122 detects an enhanced version of frame 304 (e.g., signature tone 104 and acquired audio signals (e.g., which correspond to recording signal 300, audio data 302, and frame 304)) to detect the start time of frame 304. GCC PHAT block 112 then receives frame 304 as input.

[0052] refer to Figure 1 and Figure 4 , the GCC PHAT block 112 processes the output of the signature frame detection block 110 to provide an estimated DOA of the captured audio signal transmitted by at least the speaker 102. An example of the operation of the GCC PHAT is described in "The Generalized Correlation Method for Estimation of Time Delay," IEEE Transactions on Acoustics, Speech, and Signal Processing 24, No. 4 (1976): 320-327, which is incorporated herein by reference in its entirety. As described above, the speaker 102 in the system 100 can provide a mobile device 150 (see Figure 1 ) provides (or wirelessly transmits) estimated DOA readings. The mobile device 150 may be a mobile phone, a laptop, a desktop, etc. As also described above, the speakers 102 in the system 100 may include one or more transceivers 152 to wirelessly transmit and receive information (including estimated DOA readings) to each other and / or to the mobile device 150.

[0053] The GCC PHAT block 112 may utilize single path wave propagation of sound waves from a single sound source signal s(n) provided by a sound source (or any one of the speakers 102). The microphones 106a and 106b receive the signal s(n) as a received signal x 1 (n) and x 2 (n), the received signal is a delayed and attenuated version of the original sound signal s(n). In general, the controller 122 can find x 1 (n) and x 2 The maximum value of the cross-correlation of (n) is used to determine the received signal x 1 (n) and x 2 (n) The controller 112 performs the cross-correlation by executing the following equation:

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] Equations 3 to 7 are used in the GCC PHAT block 112 to estimate the sampling delay Equation 3 represents x 1 (n) and x 2 (n). Equation 4 is the cross power density, which is obtained by taking x 1 (n) and x 2 The PHAT processor (of the GCC PHAT block 112) is shown in Equation 5. Applying the inverse Fourier transform yields x as shown in Equation 6. 1 (n) and x 2 Finally, by finding x in Eq. 1 (n) and x 2 (n) to calculate the sampling delay

[0060] At this time, the controller 122 may determine the DOA of the received audio signal or the angle of the sound source 102a (or the first speaker 102a). For example, the controller 122 may determine the DOA (or angle information, “angle”) of the audio signal received by the receiving speaker 102 by:

[0061]

[0062] in is an estimate of the sampling delay as described above, c is the speed of sound, and d is the distance between microphones 106a and 106b, which is a known value. GCC Phat block 112 estimates the phase difference between the audio captured between microphones 106a and 106b. Therefore, the phase difference generally corresponds to as shown in Equation 8 (or angle information). The controller 122 converts the phase difference into an enable as described in Equation 8, in particular using arc cosine. The controller 122 determines the sampling delay The way in Figure 4 Shown in.

[0063] Figure 5 Various signature tone signals 500, 502 are shown according to one embodiment. In general, the signature tone 500 includes energy below 4kHz. The signature tone 500 can be generated based on an exponential sine sweep (ESS). In this case, it may be more desirable to provide a signature signal that includes more energy at high frequencies in order to perform the estimated DOA in a higher noise based environment. The signature tone signal 502 is generated based on an inverse ESS. Figure 5The signature tone signals 500, 502 are shown to be both in the frequency domain. The signature tone signal 502 has a higher amplitude after 1kHz, which prevents the signature tone signal 502 from being distorted by background noise. In one example, the signature tone 104 generated by the first speaker 102a or the second speaker 102b may be based on an inverse ESS from a predetermined frequency range, which may be 700Hz to 10kHz and have a predetermined length of, for example, 150ms at a predetermined frequency of, for example, 48kHz. The disclosed system 100 generally provides accurate DOA estimation for speaker automatic calibration under low signal-to-noise ratios and reverberation. These aspects can enable immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner situations (such as two speakers, for example, in the same line but not aligned with each other). It should be appreciated that the signature tone signals 500, 502 (e.g., inverse EES signals) have been found to withstand high noise environments at least at a level of -10dB. For example, it has been found that the inverse ESS signal is unaffected in high noise environments of at least −10 dB, which is used to provide a sufficient signal to trigger automatic calibration of the various speakers 102 in the system 100 and determination of DOA.

[0064] Optimization of automatic speaker calibration and speaker configuration layout estimation

[0065] As described above, the speakers 102 in the system 100 are configured to communicate with each other. Each of the first speaker 102a and the second speaker 102b can provide high audio quality while utilizing immersive sound. The immersive sound technology depends on the positions of the first speaker 102a and the second speaker 102b. Therefore, in this regard, the immersive sound technology requires an automatic speaker calibration process.

[0066] There are multiple ways to perform automatic calibration. One way to perform automatic calibration requires providing an estimate of the speaker azimuth, also known as DOA. The DOA of the audio signal transmitted from each speaker can be detected by playing a signature tone from one speaker at a time. Then, the angles (or DOA) from different speakers are used to create the speaker configuration in the room. In some cases, due to environmental conditions and the position of the speakers, the estimate of the azimuth obtained may be wrong. Such errors may mainly occur when the speakers are not aligned with each other (for example, due to speaker directivity) and the background noise has more energy than the signature tone. Since these aspects may occur in real-life scenarios, the automatic calibration technology implemented in the speakers should be able to handle these scenarios. The system 100 disclosed herein provides multiple DOA estimates for optimizing speaker positions and estimating speaker layout configurations of two or more speakers. The system 100 also provides an accurate representation of the speaker configuration required for a true immersive experience. The disclosed embodiments can improve robustness and overcome the above-mentioned environmental conditions. In addition, the disclosed embodiments may provide (i) accurate speaker configuration estimation, (ii) speaker orientation estimation, (iii) detection of DOA estimation outliers while taking into account background noise, reverberation, and obstacles, and (iv) optimization of speaker configuration estimation based on previous DOA estimation and outlier detection.

[0067] Return to reference Figure 1 , the system 100 also includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. The matrix block 114 stores the DOA estimates of each of the first speaker 102a and the second speaker 102b. As described above, it should be appreciated that the system 100 may include any number of speakers 102, but preferably, more than two speakers 102 may be required. The microphone orientation estimation block 116 estimates the orientation of each of the microphones 106a and 106b positioned on the speaker 102. The outlier detection block 118 detects outliers that may exist in the matrix formed by the matrix block 114. These outliers or errors can be attributed to erroneous DOA estimates or obstacles between the first speaker 102a and the second speaker 102b. The optimization block 120 performs reference microphone selection, initial layout estimation, candidate coordinate estimation, and optimal coordinate selection. These aspects will be discussed in more detail below.

[0068] Figure 6 A method 600 for performing optimized speaker auto-calibration and speaker configuration estimation is shown according to one embodiment.

[0069] In operation 602, the microphone orientation estimation block 116 estimates the orientations of the microphones 106a and 106b. Figure 8 and Fig. 9 This operation is discussed in more detail.

[0070] In operation 604, the outlier detection block 118 detects outliers that may exist in the matrix formed by the matrix block 114 with respect to the DOA. Figure 10 to Figure 11 This operation is discussed in more detail.

[0071] In operation 606, the optimization block 120 performs reference microphone selection. Fig.12 This operation is discussed in more detail.

[0072] In operation 608, the optimization block 120 uses the DOA estimate to perform an initial layout estimate. Fig.13 This operation is discussed in more detail.

[0073] In operation 610, the optimization block 120 calculates candidate coordinate estimates. Figures 14 to 16 This operation is discussed in more detail.

[0074] In operation 612, the optimization block 120 selects the best coordinates. Figure 17 to Figure 18 This operation is discussed in more detail.

[0075] Figure 7 An example of a speaker and microphone configuration 700 in system 100 according to one embodiment is shown. Configuration 700 includes Figure 1 The speaker 102. Reference Figure 7 and below, Figure 1 The speakers 102 are generally shown as a first speaker 102a, a second speaker 102b, a third speaker 102c, and a fourth speaker 102d. Figure 1As described above, any number of speakers may be provided. Each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a first microphone 106a and a second microphone 106b. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a controller 122, a memory 130, and a transceiver 152. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a TF masking block 108, a signature frame detection block 110, and a GCC PHAT block 112. Also as described above, the mobile device 150 includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. It should also be recognized that in another embodiment, the system 100 may include a main speaker 103. The primary speaker 103 may correspond to any one of the speakers 102a to 102d and may be simply designated as the primary speaker to perform similar tasks as the mobile device 150. For example, the primary speaker 103 may be arranged to provide a layout of the speakers 102, including a layout of the primary speaker 103 based on the principles disclosed herein. In this sense, if it may be preferred that the primary speaker 103 provides the positions of the respective speakers 102 and 103 within the listening environment 151 in order to establish channel assignments for the speakers 102 and 103, the primary speaker 103 provides a similar level of functionality as that provided in conjunction with the mobile device 150. Thus, in this regard, the primary speaker 103 may include a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. Although the primary speaker 103 may provide the positions of the speakers 102, 103 in the listening environment 151 in a manner similar to that explained using the mobile device 150, the primary speaker 103 may not provide any visual indicators or prompts to the user regarding the positions of the speakers 102, 103.

[0076] The first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d wirelessly communicate with each other and / or with the mobile device 150 via the transceiver 152 to provide a layout of speakers in the listening environment 151. In particular, the mobile device 150 can provide a layout of the various speakers 102a, 102b, 102c, and 102d arranged in the listening environment 151. In general, the specific layout of the speakers 102a-102d relative to each other may be unknown, and the various aspects set forth herein can determine the specific layout of the speakers 102a-102d in the listening environment 151. Once the layout of the speakers 102a-102d is known, the mobile device 150 can assign channels to the speakers 102a-102d in a determined manner based on a pre-stored or predetermined system configuration.

[0077] The mobile device 150 may display the layout of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d based on the information received from these devices. In one example, the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d may wirelessly transmit DOA estimates, microphone orientation estimation information, outlier information, reference speaker selection information, initial speaker layout estimates, candidate coordinate estimation information, and optimal coordinate selection information to each other via the transceiver 152 and / or with the mobile device 150, as described in method 600.

[0078] A legend 702 is provided, which shows various angles of the position of microphones 106a to 106b on one speaker 102 relative to the microphones 106a to 106b on other speakers 102a, 102b, 102c, and 102d. Various operations of method 600 will be described below with reference to legend 702. The first speaker 102a, the third speaker 102c, and the fourth speaker 102d show that their respective microphones 106a to 106b are arranged horizontally on such speakers 102a, 102c, and 102d. The second speaker 102b shows that the microphones 106a to 106b are arranged vertically on the second speaker 102b. It should be recognized that the arrangement of the microphones 106a to 106b is unknown before determining the speaker layout, and the arrangement of the microphones 106a to 106b can be arranged in any number of configurations on the speakers 102a to 102d in the listening environment 151. The disclosed system 100 and method 600 are configured to determine a speaker configuration layout while taking into account different configurations of the microphones 106a - b .

[0079] Referring to the first speaker 102a and further referring to the legend 702, the first speaker 102a captures audio (or detects audio) from the second speaker 102b at 0 degrees. The first speaker 102a captures audio (or detects audio) from the third speaker 102c at 45 degrees. The first speaker 102a captures audio from the fourth speaker 102d at a 90 degree angle. Figure 7The angles (or angle information) at which the remaining speakers 102b to 102d receive audio relative to the other speakers 102a to 102d are shown. Any reference to the term "angle" may also correspond to "angle information" and vice versa. The relevance of the angles (or angle information) will be discussed in more detail below. It should be appreciated that each of the speakers 102a to 102d transmits information related to the angle information at which they receive audio from each other to the mobile device 150 or other suitable computing device. The mobile device 150 stores the angles in its memory. The DOA information reported by the speakers 102a to 102d is reported as the angles mentioned above.

[0080] Figures 8 to 9 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Example of microphone orientation and operation 604 of method 600. At least one embodiment provides a dual speaker position in space, which establishes a line, and the slope of the line does not change when viewed from one speaker or another. In general, the system 100 and / or method 600 recognizes that the dual speaker position in space establishes a line, and the slope of the line does not change from one speaker to another. The first matrix 800 shown depicts the relative angle of the audio received relative to each speaker 102a to 102d (or S1 to S4, respectively). Any angle reading of -360 indicates a null value. For example, the first matrix 800 shows that the S1 in the columns and rows of the matrix is ​​-360 because the first speaker 102a (or S1) cannot receive audio from itself. Any value of 360 for the angle of the second speaker 102b (or S2), the third speaker 102c (or S3) and the fourth speaker 102d (or S4) further illustrates this point.

[0081] The mobile device 150 generally stores information corresponding to the angle information shown in the first matrix 800. The first column as shown by the dashed box in the first matrix 800 corresponds to a specific speaker that is receiving audio from the speakers S1 to S4 shown in columns 2 to 5, respectively. For example, referring to the first column and the second row, the second speaker (e.g., or S2) 102b receives audio from the first speaker (e.g., or S1) 102a (as shown in the second column) at a 90-degree angle, the second speaker 102b receives audio from the third speaker 102c at a 0-degree angle, and the second speaker 102b receives audio from the fourth speaker 102d (or S4) at a 45-degree angle. Referring to the first column and the third row, the third speaker 102c (e.g., or S3) receives audio from the first speaker 102a (e.g., or S1) at a 45-degree angle, and the third speaker 102c (e.g., or S3) receives audio from the fourth speaker 102d (e.g., or S4) at a 0-degree angle. Referring to the first column and the fourth row, the fourth speaker 102d receives audio from the first speaker 102a (or S1) at a 90 degree angle, the fourth speaker 102d receives audio from the second speaker 102b (or S2) at a 135 degree angle, and the fourth speaker 102d receives audio from the third speaker 102c (or S3) at a 180 degree angle.

[0082] See also Fig. 9 , the mobile device 150 receives information corresponding to each angle from the transceiver 152 of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. As described above, the mobile device 150 assembles the first matrix 800 based on the information received from the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. The mobile device 150 may determine the orientations of the microphones 106a to 106b of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to each other. In particular, the mobile device 150 may determine whether the orientations of the microphones 106a to 106b of the respective first speakers 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are different from each other based on the angles stored in the first matrix 800. The embodiments disclosed herein generally illustrate that the slope may not change, but the angle depends on the orientation of the microphones 106a - 106b , which may be from the first row and first column generally shown at 900 .

[0083] For example, the mobile device 150 may determine whether the angle differences between the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d shown in the first matrix 800 correspond to one or more predetermined values ​​(e.g., 0 or 180). If the differences between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d correspond to one or more predetermined values, the mobile device 150 may determine that the microphones 106a to 106b of the two or more speakers 102a, 102b, 102c, 102d are in the same orientation. If the differences between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d do not correspond to one or more predetermined values, the mobile device 150 may determine that the microphones 106a to 106b of the two or more speakers 102a, 102b, 102c, 102d are not in the same orientation.

[0084] refer to Fig. 9 In the first matrix 800 shown, the mobile device 150 determines that the second speaker 102b has a 90 degree difference relative to the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. Therefore, in this regard, the mobile device 150 determines that the orientation of the microphones 106a-106b of the second speaker 102b is different from the orientation of the microphones 106a-106b of the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. This Fig. 9 Shown in.

[0085] Generally, the mobile device 150 subtracts the angle of the first column from the first row to perform microphone orientation estimation. Then, a subtraction operation is performed, and the result is [0, 90, 0, 0] for the first speaker 102a (or S1), the second speaker 102b (or S2), the third speaker 102c (or S3), and the fourth speaker 102d (or S4). Therefore, the microphone estimation of the third speaker 102c (S3) and the fourth speaker 102d (S4) is 0, which is the same as the orientation of the first speaker 102. The mobile device 150 can also perform microphone orientation using a modulo operation after performing the subtraction operation, because the angle range should be [0, 180], such as Figure 7 702 of FIG. In general, the slope between the speakers 102a to 102d may not change, but the angle may depend on the orientation of the microphones 106a to 106b. The mobile device 150 generates a microphone orientation array 900 that includes an angle difference that does not correspond to 0 or 180 degrees (or a predetermined value). The embodiments disclosed herein generally show that the slope may not change, but the angle depends on the orientation of the microphones 106a to 106b, which can be found from the first row and the first column of the microphone orientation array.

[0086] Figures 10 to 12 According to one embodiment, Figure 7 Executed on the configuration Figure 6 An example of an outlier detection operation 604 of method 600. After performing the microphone orientation estimation of operation 602, the mobile device 150 performs an outlier detection operation 604 to determine whether any of the speakers 102a-102d is an outlier with respect to the layout. If any of the speakers 102a-102d is determined to be an outlier, the mobile device 150 determines that the position of the speaker 102 relative to the positions of the other speakers 102a-102d is incorrect or cannot be determined.

[0087] refer to Fig.10 As part of operation 604, the mobile device 150 subtracts the microphone orientation array 900 from the first matrix 800 to provide a calibrated angle matrix 902. The mobile device 150 takes into account speakers whose microphone orientations are not aligned with the rest of the microphones of the speaker. Fig.10 The calibration angle matrix 902 shown shows that the 90 degree angles from the microphone orientation array 900 have been subtracted from the angles in the second row (S2) (except -360 as this is a null value) to provide the following in the calibration angle matrix 902 [0, -360, 90, 135].

[0088] refer to Fig.11 According to one example, as part of operation 604, the mobile device 150 may compare the angles shown in row 2 of the calibration angle matrix 902 to a predetermined threshold. If any one or more angles in row 2 of the calibration angle matrix 902 are above the predetermined threshold, the mobile device 150 detects an outlier of one or more speakers 102a to 102d having an angle above the predetermined threshold. The mobile device 150 generates a block matrix 1100, such as Fig.11 As shown generally. For example, the mobile device 150 checks the difference between each pair of estimates (e.g., the angle estimate of the second speaker 102b at the first speaker 102a and the angle estimate of the first speaker 102a at the second speaker 102b). For example, the mobile device 150 may apply modulo 180 to ensure that the difference is within the range of [0, 180] degrees. If the difference is above a predetermined threshold, the mobile device 150 may determine that an outlier exists for the pair of speakers. In this example, the block matrix 1100 does not indicate an error in any of the speakers 102a to 102d. Therefore, in this regard, the angle in row 2 of the calibration angle matrix 902 is less than the predetermined threshold. Outliers generally represent various erroneous estimates in the DOA matrix. Detected outliers may not be used for optimization (e.g., operations 606, 608, and 610).

[0089] refer to Fig.12 And similar to combining Fig.11 According to another example, as part of operation 604, mobile device 150 may compare the angle shown in row 2 of calibration angle matrix 902 with a predetermined threshold. Fig.12 In the example shown, row 2 of the calibration angle matrix 902 is different from Fig.11 , and corresponds to [0, -360, 25, 135]. In this regard, when the mobile device 150 compares the angle shown in row 2 of the calibration angle matrix 902 to a predetermined threshold, the mobile device 150 generates a value of "1", as shown in row 3, column 2 and row 2, column 3 in the block matrix 1100. Referring to the block matrix 1100, all values ​​of 0 do not represent outliers, and as described above, an angle of -360 corresponds only to a null value and can be ignored. In this case of the block matrix 1100, the third speaker 102c is determined to be an outlier relative to the second speaker 102b. The estimates in row 2, column 3 and row 3, column 2 will not be used in operations 606, 608, 610.

[0090] Fig.13 According to one embodiment, Figure 7 Executed on the configuration Figure 6 An example of reference speaker selection operation 606 of method 600. Mobile device 150 may then examine any rows / columns of block matrix 1100 that are filled with "1". As described above, these values ​​typically indicate that the speaker is an outlier. Fig.13 The block matrix 1100 shown is similar to the combination of Fig.11 The blocking matrix 1100 is shown and does not indicate the presence of any outliers. If the mobile device 150 does not detect an outlier in the blocking matrix 1100, the mobile device 150 generates an error and repeats the method 600 again.

[0091] Fig.14 and Fig.15 According to one embodiment, Figure 7 Executed on the configuration Figure 6 An example of microphone orientation and speaker layout estimation operation 608 of method 600. Fig.14, the configuration of the first, second, third and fourth speakers 102a to 102d shown is now reflected as including distance coordinates in the x and y axes. As shown, the first speaker 102a is selected as the reference speaker. The second speaker 102b has coordinates (100, 0) relative to the first speaker 102a, the third speaker 102c has coordinates (70.71, -70.71) relative to the first speaker 102a, and the fourth speaker 102d has coordinates (0, -100) relative to the first speaker 102a. As described above, the mobile device 150 does not know the exact layout of the speakers 102a to 102d in the listening environment 151. In this way, the mobile device 150 establishes a reference matrix 1400 with reference coordinates (or distances or values): 0, 100, 100, 100, for the first speaker 102a, the second speaker 102b, the third speaker 102c and the fourth speaker 102d, respectively. The mobile device 150 randomly selects coordinates (e.g., 0, 100, 100, 100). In this case, the mobile device 150 assumes that the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are located equally far from the first speaker 102a. As shown in the first matrix 800, the mobile device 150 has information corresponding to angles relative to the audio received by the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d. However, the actual distances of such speakers 102a to 102d are unknown.

[0092] Reference now Fig.15 , the mobile device 150 calculates the distances (or x, y coordinates) of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a, which can be determined based on the following equation 9:

[0093] (x,y)(x s1 +Distance s1sj *cos(DOA s1sj ),y s1 -distance s1sj *sin(DOA s1sj )

[0094] For example, the mobile device 150 may calculate the distance coordinates of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a based on equations 10, 11, and 12, respectively:

[0095] (x 2 ,y 2 )=(100*cos(0),-100*sin(0))=(100,0)

[0096] (x3 ,y 3 )=(100*cos(45),-100*sin(45))=(70.71,-70.71)

[0097] (x 4 ,y 4 )=(100*cos(90),-100*sin(90))=(0,-100)

[0098] Equation 10, as shown above, corresponds to the distance coordinate of the second speaker 102b relative to the first speaker 102a, where the angle 0 is inserted into Equation 5 and is taken from the first row (i.e., S1) and the second column (i.e., S2) of the first matrix 800. Equation 11, as shown above, corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle 45 is inserted into Equation 8 and is taken from the first row (i.e., S1) and the third column (i.e., S3) of the first matrix 800. Equation 11, as shown above, corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle 90 is inserted into Equation 12 and is taken from the first row (i.e., S1) and the third column (i.e., S3) of the first matrix 800.

[0099] Figures 16 to 20 According to one embodiment, Figure 7 Executed on the configuration Figure 6 Various aspects of the candidate coordinate estimation operation 610 of method 600 . Fig.16 Generally, it is shown that based on the performance of operation 608, the layout estimate of the third speaker 102c is positioned at the coordinates (70.71, -70.71) relative to the first speaker 102a. However, the actual layout indicates that the third speaker 102c is positioned at the coordinates (70.71, -70.71) relative to the first speaker 102a, the third speaker 102c is positioned at the coordinates (100, -100) relative to the second speaker 102b, and the third speaker 102b is positioned at the coordinates (-100, -100) relative to the fourth speaker 102d. These aspects are generally shown as candidate coordinate estimates 1600. Fig.17 Shown according to the combination Fig.15The discussed equations 10, 11, and 12 determine the manner in which the various coordinates of the third speaker 102c are determined relative to the first, second, and fourth speakers 102a to 102d. In general, the mobile device 150 does not know whether the coordinates of the third speaker 102c are correct. In this case, the mobile device 150 estimates possible candidate points. In operation 612, the mobile device 150 calculates the error for each candidate point. The candidate point that exhibits the lowest error is selected as the best coordinate. It should be recognized that all DOA estimates from all speakers 102 are transmitted to the mobile device 150 using any number of wireless communication protocols (such as but not limited to Bluetooth, WiFi, etc.). In Fig.17 In the example shown, the mobile device 150 uses angles from the calibration angle matrix 902 to determine the coordinates of the third speaker 102c relative to the first speaker 102a, the second speaker 102b, and the fourth speaker 102d.

[0100] refer to Fig.18 , the mobile device 150 expands the candidate coordinate estimate 1600 by combining the x and y points. As generally shown at 1800, in addition to the expanded candidate coordinate estimate 1802, the candidate coordinate estimate 1600 is provided. In general, the candidate coordinate estimate reflects the x and y coordinates in the following manner: the first speaker 102a, the second speaker 102b, and the fourth speaker 102d are (x a ,y a )、(x b ,y b ) and (x c ,y c The expanded candidate coordinate estimates 1802 reflect the x and y coordinates in the following manner: the first speaker 102a and the second speaker 102b and the fourth speaker 102d and the first speaker 102a are respectively (x a ,y b )、(x c ,y z). The expanded candidate coordinate estimate 1802 is expanded in the manner shown at 1800 because some degree estimates provide information in one dimension (e.g., x coordinate, y coordinate), for example. The mobile device 150 combines the coordinates to obtain information in a two-dimensional format (e.g., x coordinate and y coordinate). The mobile device 150 typically combines the candidate coordinate estimate 1600, the expanded candidate coordinate estimate 1802, and the expanded angle as shown at 1804. The expanded angle 1804 is typically estimated using the angles (e.g., S2 and S1) of the first speaker 102a and the second speaker 102b corresponding to (100, -100). The first speaker 102a and the third speaker 102c form a line, and the coordinates (100, -100) are calculated using the intersection of the two lines. The angle information is used to form the line. The mobile device 150 expands the candidate coordinate estimate to locate the intersection between the third speaker 102c and the second speaker 102b and the fourth speaker 102d. The mobile device 150 does not consider the candidate coordinate estimate 1600, the expanded candidate coordinate estimate 1802, and the expanded angle 1804 of the third speaker 102c because the position of the third speaker 102c is incorrect. In general, the mobile device 150 does not need to determine whether any position is correct. The figures described herein are provided only as examples. Fig.18 The operations of the third speaker 102 c are disclosed as an example. Throughout the method, these operations apply to each speaker 102 in the system 100 .

[0101] refer to Fig.19 , the mobile device 150 continues to expand the candidate coordinate estimate 1600. In this case, a combination of Fig.19 The example shown is to simply illustrate another example of expanding the candidate coordinates, and combined with Fig.19 The examples shown may be independent of the examples shown above.

[0102] Fig. 20 Another example of a candidate coordinate estimation operation 610 contemplated by mobile device 150 is shown. Fig. 20Another example of a modified first matrix 800' and a modified blocking matrix 1100' is shown. The modified first matrix 800' shows that there is an obstacle between the fourth speaker 102d and the third speaker 102c, as shown by the angle "145" in the third column (e.g., S3) and the fourth row (e.g., S4). In general, for a four-speaker layout configuration, the method 600 can only tolerate a single outlier between any two speakers. In this case, the mobile device 150 determines two or more outliers (e.g., the third speaker 102c and the fourth speaker 102d). As discussed above in conjunction with operation 604, the mobile device 150 determines that the third speaker 102c is an outlier. In this regard, the mobile device 150 also determines that the fourth speaker 102d is an outlier based on the description provided above in conjunction with operation 604. In view of the presence of more than one outlier, the mobile device 150 includes a user interface and commands the user to move any obstacles that exist relative to the third speaker 102c and the fourth speaker 102d. Given that the third speaker 102c and the fourth speaker 102d are outliers (i.e., an obstacle is formed therebetween), the mobile device 150 does not consider the candidate coordinate estimates 1600 (or vice versa), the expanded candidate coordinate estimates 1802, and the expanded angles 1804 from the third speaker 102c and the fourth speaker 102d because there is an outlier between the positions of the third speaker 102c and the fourth speaker 102d, and such estimates are considered incorrect. The mobile device 150 updates the modified blocking matrix 1100', which illustrates that the third speaker 102c and the fourth speaker 102d are blocked when considering the layout.

[0103] Fig.21 and Fig. 22 According to one embodiment, Figure 7 Executed on the configuration Figure 6 An example of the best coordinate selection operation 612 of the method. The mobile device 150 performs the best coordinate selection operation 612. For example, the mobile device 150 executes the following equation (13):

[0104]

[0105] in is the angle calculated by using the candidate x and y coordinates of the ith speaker, and C corresponds to the candidate index. The mobile device 150 selects the candidate point that minimizes the error. The calibration DOA matrix 800 set forth above is used as the DOA matrix in the above equation. ij .

[0106] Fig. 22 An example of the manner in which equation 13 is implemented is generally shown. It should be appreciated that if Fig. 22 shown It can be obtained by the following equation:

[0107]

[0108] The mobile device 150 determines the errors of the third speaker 102c and the first speaker 102a respectively based on Equation 13:

[0109] Error 31 = |45-45| + |90-150| + |0-20| = 80

[0110] Similarly, the mobile device 150 also determines the errors of the third speaker 102c and the second speaker 102b respectively based on Equation 10:

[0111] Error 32 = |45-45| + |0-0| + |180-180| = 0

[0112] Fig. 22 It is shown that it can be calculated using Equation 10 Fig. 22 The first two rows of the table are used to estimate the position of the third speaker. From the second term in Equation 5. Fig. 22 As shown in the table represents the angle between the speaker 102 and the candidate point.

[0113] Fig.23 An example of a microphone orientation and speaker estimation method 600 is shown according to one embodiment. Fig.23 is the position of the speakers 102a to 102d in the listening room 151 (or the listening environment). The mobile device 150 may display the positions (e.g., front, left, right, and rear) of the speakers 102a to 102d in the listening room. In general, the system 100 and / or the method 600 are based on at least Figures 6 to 22 The method shown determines the positions of the first, second, third, and fourth speakers 102a to 102d in the listening environment 151. The system 100 and / or method 600 utilizes the position information to provide channel assignment for immersive sound generation of the speakers 102a to 102d. For example, the mobile device 150 utilizes the final DOA to assign the speakers 102a to 102d to the front, left, right, and rear speaker positions.

[0114] System and method for boundary distance estimation

[0115] As described above, the first, second, third, and fourth speakers 102a to 102d are typically from a family of products, all of which are equipped with microphones 106a to 106b mounted thereon. The microphones 106a to 106b of each speaker 102 provide the ability to detect the location of an audio source (e.g., a mobile device 150) relative to any nearby walls. However, since the microphones 106a to 106b may be arranged linearly when packaged on the corresponding speakers 102, the microphones 106a to 106b may lack the ability to distinguish between audio sources in front of or behind the speakers based on using the line between the microphones 106a to 106b as a line of symmetry. Detecting a wall or obstacle in one of the directions eliminates the symmetry limitation.

[0116] Moreover, if the speaker is placed too close to a wall or a corner, the speaker may not be detected. The disclosed system can detect whether the speaker is placed too close to the wall, and automatically correct the speaker positioned close to the wall to ensure that the desired sound field is transmitted in the room (or listening environment 151). In general, speakers close to the wall will have a + / -3dB effect at low frequencies. In addition, the disclosed system and method can be used for improved audio upmixing. The various aspects disclosed herein can provide (for example) a circular microphone array with six microphones, which can detect all surrounding walls using the disclosed method. At this time, the disclosed method can determine whether the surrounding wall of the microphone is the left wall or the right wall by comparing the proximity of the wall to each microphone. At this time, the system can perform channel allocation that can be used for upmixing that can be automatically performed. In addition, the disclosed system and method can obtain room characteristics and estimate the distance to the wall or reflector.

[0117] The room impulse response (RIR) generally provides an audio fingerprint of a location in an acoustic environment. RIR can have a variety of applications, such as wall boundary estimation, digital reconstruction of acoustic environments in professional audio applications, room correction, and frequency response correction of playback systems. The measurement of RIR includes exciting the room (or listening environment), which can be performed by, but not limited to, clapping. The measurement of RIR may also include deconvolution of the audio signal to obtain room characteristics. RIR may involve reflections after exciting the room. Reverberation may refer to audio reflections reflected back to the audio source. Reverberation is usually not a direct sound, so the reverberation arrives at the microphone later. The reflection amplitude and return time depend on the material of the surface and the number of reflective areas. The sound continues to reflect until the sound loses its energy due to absorption.

[0118] Fig.24A system 2400 for performing boundary estimation according to one embodiment is shown. The system 2400 generally includes a first speaker 102a and a second speaker 102b. Each of the speakers 102a, 102b generally includes an audio source 2402. Although only a first speaker 102a and a second speaker 102b are shown, it should be appreciated that any number of speakers may be positioned in a listening environment. The audio source 2402 may be integrated into any of the speakers 102a, 102b to transmit audio from a particular speaker 102 directly into the listening environment 151.

[0119] The first speaker 102a and the second speaker 102b are located away from the wall 2404. In general, it is desirable to know the distance of the first speaker 102a and / or the second speaker 102b from the wall 2404 in the listening environment 151. If one or more of the first speaker 102a and / or the second speaker 102b are placed too close to the wall 2404, such a situation may make it difficult for the audio source 2402 to automatically correct the position of the wall 204 relative to the speakers 102a to 102b to ensure that the desired sound field is transmitted into the room (or listening environment 151). In general, the first speaker 102a and / or the second speaker, if positioned too close to the wall 2404, may cause a + / - 3dB effect at low frequencies, respectively. The audio source 2402 (i.e., within the speaker 102a and / or the speaker 102b) can determine the position of the first speaker 102a and / or the second speaker 102b relative to the wall 2404 and employ a correction mechanism to account for the distance of the first speaker 102a and / or the second speaker 102b positioned close to the wall 2404. The system 2400 can improve channel assignment using more than two microphones 106a by employing a correction mechanism to address the close proximity of the speakers 102a-102b to the wall 2404. The ability to perform channel assignments (e.g., which speaker is left front / right front / rear, etc.) properly enables audio upmixing. It should be appreciated that the audio source 2402 can include any number of controllers 2410 (hereinafter referred to as "controllers 2410") to perform the operations described herein. Although the audio source 2402 can determine the distance of the first speaker 102a and / or the second speaker 102b relative to the wall 2404, it should be recognized that any one or more of the first speaker 102a or the second speaker 102b can also include at least one controller 2412 to determine the distance of the speakers 102a, 102b relative to the wall 2404.

[0120] The controller 2410 may provide and transmit an audio fingerprint in the listening environment 151 using, for example, a predetermined measurement scheme such as RIR. For example, the controller 2410 may include a driver (not shown) to transmit the audio fingerprint into the listening environment 151. The controller 2410 may also include a memory for storing the audio fingerprint. The system 2400 may employ various applications of RIR, such as wall boundary estimation, digital reconstruction of acoustic environments for professional audio applications, room correction, and frequency response correction for playback systems. In one example, the audio source 2402 may excite the room (or listening environment 151) by transmitting an audio signal, and the measurement of the RIR may also include deconvolution of the audio signal to obtain room characteristics. As described above, the RIR may involve measuring the captured audio fingerprint (i.e., reflection) after the listening room 151 has been excited. Reverberation may refer to audio reflections reflected back to the audio source 2402. The audio source 2402 may be coupled to microphones 106a and 106b to receive captured reflections (or reverberation) from the listening environment 151. The reverberation received back by the audio source 2402 is generally not a direct sound, so the reverberation arrives at the microphone 106 later. The amplitude of the reverberation and the time it takes for the reverberation to arrive at the audio source 2402 depends on the number of surface materials and reflective areas within the listening environment 151. The sound continues to reflect until the sound loses its energy due to absorption within the listening environment 151.

[0121] The audio source 2204 may excite the listening environment 151 by transmitting an audio signal comprising an exponential sine sweep (ESS) (or an ESS signal). The ESS signal may be advantageous over the impulse response measurement method because (i) for a signal transmitted with the same length as the length of a maximum length sequence (MLS), the ESS signal has better noise suppression than the MLS method, and (ii) given that the driver transmits the ESS signal directly, the ESS signal may be more robust to nonlinear effects.

[0122] The following equation can be given for the ESS signal:

[0123]

[0124] in:

[0125]

[0126] T represents the duration of the scan. The variable ω 1 and ω 2 Corresponding to the starting and ending frequencies respectively. Due to the frequency variation of the ESS, the energy may depend on the rate of the instantaneous frequency as follows:

[0127]

[0128] Fig.25Generally shown is a frequency response of an ESS signal 2500. When the signal 2500 is transmitted from the audio source 2402 to one or more of the first speaker 102a and the second speaker 102b, the ESS signal 2500 includes a peak 2502 thereof.

[0129] The audio source 2402 may measure the RIR by inverse filtering or deconvolution after the first speaker 102a and / or the second speaker 102b plays the EES signal 2500 in the listening environment 151. Then, the controller 2410 applies inverse filtering and extracts the RIR. As described above, the audio source 2402 includes any number of microphones 2420 to record the ESS signal 2500. Then, the audio source 2402 may extract or measure the RIR from the recorded ESS signal 2500. The time-reversed energy of the ESS signal 2500 is reduced, for example, by 3 dB / octave, and the inverse filter has, for example, a 3 dB / octave increase in its energy spectrum to obtain a flat frequency spectrum. Assume that h(t) is the room impulse response, r(t) is the excitation room impulse response, and f(t) is the inverse filter.

[0130] h(t)=r(t)*f(t) (13)

[0131] f(t) can be created using post-modulation, which applies an amplitude modulation envelope of +6dB / octave to the spectrum of the time-reversed signal. The general form of the post-modulation function is as follows:

[0132]

[0133] A represents the constant of the modulation function. For time t = 0, ω(t) = w 1 , and for unity gain at time ι = 0:

[0134]

[0135] Then, the modulation function becomes:

[0136]

[0137] After modulating the time-reversed signal with m(t), f(t) now has a frequency increase of 3dB / octave. Fig.26 The amplitude spectrum of the inverse filter is shown.

[0138] In general, the measured RIR is obtained by the audio source 2402 by utilizing Equation 13. Thus, aspects associated with Equation 13 correspond to the convolution of the ESS signal and the inverse filter. The audio source 2402 may utilize the measured RIR to estimate the distance of the first speaker 102a and / or the second speaker 102b to the wall 2404. It should be appreciated that the audio source 2402 given the speakers 102a and 102b determines the distance of each speaker 102a and 102b in which the audio source 2402 is positioned. In general, the wall proximity estimate used by the audio source 2402 may be complex because the measured RIR includes reverberation from multiple objects in the listening environment 151 (or room).

[0139] Fig. 27 An example of a curve 2700 corresponding to an RIR measurement performed by an audio source 2402 is generally shown. As shown, the RIR measurement 2700 includes a plurality of peaks 2702. The peaks 2702 may correspond to reflections or reverberations of the ESS signal from various objects in the listening environment 151. A reverberation number 1850 is generally shown at 2704. The reverberation number 1850 generally corresponds to a strong candidate for reverberation of the ESS signal from a wall 2404. Since the amplitude of the peak is highest after the gap 2710 between the highest peak 2712 shown and the reverberation number 1850 as shown at 2704, this condition may be verified. The highest peak 2172 generally indicates a direct path for the ESS signal from the speaker 102 to the microphone 2420. Additionally, since the peak amplitude associated with 2704 may correspond to the material of the wall 2404, this condition may also be verified. Generally speaking, the peak from the wall 2404 may not be obvious, as shown in FIG. Fig. 27 Additionally, the nonlinearity attributed to peak 2702 may be due to a driver (or amplifier) ​​in audio source 2402. For example, amplifiers often produce false peaks in RIR measurements performed by audio source 2402. Therefore, audio source 2402 may need to account for these conditions when performing RIR measurements.

[0140] The audio source 2402 can overcome the above problems to perform wall distance estimation by (i) sampling or extracting the peaks in the RIR measurement to avoid strong and false peaks (or ringing) close to the peak to be detected around the peak 2702 that may cause an erroneous estimate, and / or (ii) scoring each peak to determine the correct peak from the wall 2404. It should be recognized that due to nonlinearity, there are undesirable peaks around the peak 2702, and it is desirable to avoid such peaks in the RIR measurement. In general, the peak 2702 in the RIR measurement may correspond to the direct path from the audio source 2402 to the microphone 2420 and from the reflector to the microphone 2420 on the audio source 2402. Looking closely at the RIR measurement, it can be observed that there is ringing around the peak. The audio source 2402 can extract peaks to detect impulse events. Therefore, in this regard, the audio source 2402 can utilize a sliding window to extract the peak in each window. The audio source 2402 may find each peak in the window after obtaining the largest peak in the RIR measurement and ignore other peaks in the RIR measurement.

[0141] Fig.28 Generally illustrated is an RIR measurement 2800 with a peak 2802 detected by an audio source 2402, according to one embodiment. Fig.28 It is also shown that, as above combined Fig. 27 The ringing is more pronounced or noticeable. For a sampling rate of 32 kHz, when the distance from the first speaker 102a and / or the second speaker 102b to the wall is 137 cm, the audio source 2402 obtains a RIR measurement value 2800. The distance from the speaker to the wall can be obtained based on the following equation:

[0142]

[0143] For example, the “estimated peak index” as set forth above in Equation 17 generally corresponds to the estimated peak value in the RIR measurement 2800. Thus, in this regard, Fig.28 As shown, detected peak 2802a corresponds to sample 251, which may be defined as an estimated peak index. The "maximum peak index" typically corresponds to zero. Fig.28 Generally shows Fig. 27 A modified version of the RIR measurement shown in . For example, Fig. 27 The highest peak 2712 shown in the figure generally corresponds to Fig.28Similarly, the constant value "2" as set forth in Equation 17 generally corresponds to the round trip time. When the audio source 2402 plays the ESS signal, the sound (or ESS signal) contacts the wall 2404 and reflects back to the audio source 2402. Therefore, the ESS signal samples the same distance or path twice. Therefore, in this regard, it is explained why the constant value is set to 2. Fig.28 28. As shown in the RIR measurement 2800, if the audio source 2402 correctly estimates the peak at sample 251, then for a 32kHz sampling rate, the audio source 2402 determines the distance between the wall 2404 and the first speaker 102a or the second speaker 102b to be, for example, 137cm using equation 17 above.

[0144] The audio source 2402 tracks the overall trend in the peaks 2802 of the RIR measurements 2800 to estimate the peaks of the reverberation of the RIR measurements 2800. For example, if the ESS signal transmitted by the audio source 2402 does not encounter a wall 2404 or an object in the listening environment 151, then the expected trend of the peaks 2802 of the RIR measurements will illustrate or correspond to an overall decrease in the peaks (i.e., a decreasing trend). If the ESS signal transmitted by the audio source 2402 does encounter a wall 2404 or an object in the listening environment 151, then the expected trend of the peaks 2802 of the RIR measurements will illustrate a decreasing trend in the peaks 2802, followed by an increasing trend in the peaks, followed by a decreasing trend in the peaks 2802. In general, the audio source 2402 stores information corresponding to the peaks 2802 received for the RIR measurements to determine whether there is only a decreasing trend in the peaks 2802 that continuously decreases over time, or whether there is a decreasing trend in the peaks 2802, followed by an increasing peak 2802a. The audio source 2402 may then establish a confidence score calculated by using, for example, the number of negative peaks 2802 multiplied by a percentage increase, such as 1.01. The audio source 2402 may then select a predetermined number of peaks with the highest confidence score (i.e., the maximum score) or level (e.g., 20) and then locate the maximum peak among the selected peaks 2802. Such a maximum peak may correspond to a peak that exhibits the largest amplitude on the RIR measurement and may be positive after a long series of decreasing peaks. In this case, the maximum peak may be selected as a sample number (e.g., 251), which is then used by the audio source 2402 to insert into equation 17, as provided above, to find the distance of the speaker 102a or 102b from the wall 2404.

[0145] Fig.29A method 2900 for performing boundary estimation involving multiple speakers 102 is shown according to one embodiment. In operation 2902, an audio source 2402 transmits an audio signal in the form of an ESS signal into a listening environment 151. It should be appreciated that the audio source 2402 may transmit the ESS signal from each speaker 102 positioned in the listening environment 151 one at a time and perform the operations of the method 2900 for each speaker 102 to determine a distance 2404 of the speaker 102 relative to a wall. Each audio source 2402 determines the distance of its respective speaker 102a, 102b relative to the wall 2404. It should be appreciated that each speaker 102a, 102b may also transmit the distance information to a mobile device 150 or other device that may require such information so that the mobile device or other audio source may compensate the audio output to mitigate the effect of any one or more of the speakers 102a, 102b being too close to the wall 2404.

[0146] In operation 2904, the audio source 2402 receives reverberation from the listening environment 151 in response to transmitting the ESS signal. In this case, the audio source 2402 detects the peak 2802 of the reverberation in the RIR measurement 2800 and stores information corresponding to the peak 2802 in its memory. In operation 2906, the audio source 2402 performs trend tracking of the peak 2802.

[0147] In operation 2908, the audio source 2402 evaluates the stored peak value 2802 of the reverberation to determine whether there is only a decreasing trend of the peak value 2802 that continuously decreases over time in the RIR measurement, or whether there is a decreasing trend of the peak value 2802 in the RIR measurement, followed by an increasing peak value 2802a. If the audio source 2402 determines that the peak value 2802 does not increase over time, the method 2900 moves to operation 2912 and determines that the wall distance of the first speaker 102a or the second speaker 102b cannot be determined. In this case, the method 2900 can move back to operation 2902. If the audio source 2402 determines that there is an increasing peak value 2802a in the RIR measurement, the method 2900 moves to operation 2910.

[0148] In operation 2910, the audio source 2402 establishes a confidence score calculated by using, for example, the number of negative peaks 2802 multiplied by a percentage increase, such as 1.01. The audio source 2402 may then select a predetermined number of peaks with the highest confidence scores or levels (e.g., 20) and then locate the maximum peak among the selected peaks 2802. Such a maximum peak may correspond to a peak 2802a exhibiting the largest amplitude on the RIR measurement and may be positive after a long series of decreasing peaks 2802. In operation 2912, the audio source 2402 applies the maximum peak to a distance equation (e.g., Equation 17) and also applies other variables as described above in conjunction with Equation 17 to determine the distance of the first speaker 102a or the second speaker 102b relative to the wall 2404.

[0149] It should be recognized that the controller disclosed herein may include various microprocessors, integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variants thereof) and software that cooperate with each other to perform the operations disclosed herein. In addition, such controllers disclosed utilize one or more microprocessors to execute a computer program embodied in a non-transitory computer-readable medium, the computer program being programmed to perform any number of functions disclosed. In addition, the controller as provided herein includes a housing and various numbers of microprocessors, integrated circuits, and memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) positioned within the housing. The disclosed controller also includes hardware-based inputs and outputs for receiving data from and transmitting data to other hardware-based devices discussed herein, respectively.

[0150] Although the above describes exemplary embodiments, it is not meant that these embodiments describe all possible forms of the present invention. In fact, the wording used in the specification is used for description rather than limitation, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. In addition, the features of the various embodiments can be combined to form other embodiments of the present invention.

Claims

1. An audio system, wherein include: a first speaker configured to transmit an audio signal including audio data and a signature tone to a listening environment; A second speaker, the second speaker comprising: at least one controller programmed to: receiving the audio signal including the audio data and the signature tone; determining a direction of arrival (DOA) of the audio signal as received at the second speaker based at least on the signature tone; and A time-frequency masking operation is performed on the received audio signal before determining the DOA of the audio signal.

2. The audio system of claim 1, wherein the signature tone is transmitted with a predetermined length and at a predetermined frequency.

3. The audio system of claim 1, wherein the signature tone corresponds to at least an exponentially swept sine (ESS) signal having energy within a predetermined frequency range. The audio system of claim 3 , wherein the ESS signal is an inverse ESS signal.

5. The audio system of claim 1, wherein the at least one controller is further programmed to perform a cross-correlation between the audio data and the signature sound to identify frames that bound the signature sound.

6. The audio system of claim 1, wherein the second speaker comprises at least two microphones for receiving the audio signal and a memory programmed to store the audio signal.

7. The audio system of claim 6, wherein the at least one controller is further programmed to perform the time-frequency masking operation on the stored audio signal.

8. The audio system of claim 7, wherein the time-frequency masking operation is one of ideal binary masking (IBM), ideal ratio masking (IRM), complex ideal ratio masking (cIRM), and optimal ratio masking (ORM).

9. The audio system of claim 7, wherein the at least one controller is further programmed to obtain a sampling delay associated with receiving the signature tone on the audio signal between the at least two microphones.

10. The audio system of claim 9, wherein the sampling delay is based on at least a sampling frequency, a speed of sound, and a distance between the at least two microphones.

11. The audio system of claim 9, wherein the controller is further programmed to determine the DOA of the audio signal as received at the second speaker based further at least on the sampling delay.

12. An audio system, wherein include: A first speaker, the first speaker comprising: Memory; and at least one controller programmed to receive an audio signal including audio data and a signature tone from a second speaker; storing the audio signal including the signature tone in the memory; determining a direction of arrival (DOA) of the audio signal based at least on the signature tone; and A time-frequency masking operation is performed on the received audio signal before determining the DOA of the audio signal.

13. The audio system of claim 12, wherein the signature tone is transmitted from the second speaker at a predetermined length and at a predetermined frequency.

14. The audio system of claim 12, wherein the signature tone corresponds to at least an exponentially swept sine (ESS) signal having energy within a predetermined frequency range.

15. The audio system of claim 12, wherein the at least one controller is further programmed to perform a cross-correlation between the audio data and the signature sounds to identify frames that bound the signature sounds.

16. The audio system of claim 12, wherein the first speaker comprises at least two microphones for receiving the audio signal.

17. The audio system of claim 16, wherein the at least one controller is further programmed to perform a time-frequency masking operation to reduce noise on the audio signal stored in the memory.

18. The audio system of claim 17, wherein the at least one controller is further programmed to obtain a sampling delay associated with receiving the signature tone on the audio signal between the at least two microphones.

19. The audio system of claim 18, wherein the at least one controller is further programmed to determine the DOA of the audio signal based further on at least the sampling delay.

20. A computer program product embodied in a non-transitory computer readable medium, the computer program product stored in a memory and programmed and executed by at least one controller in an audio system, the computer program product comprising instructions to: receiving, at the first speaker, an audio signal from the second speaker, the audio signal comprising audio data and a signature tone; determining a direction of arrival (DOA) of the audio signal as received at the first speaker based at least on the signature tone; and Prior to determining the DOA of the audio signal, a time-frequency masking operation is performed on the received audio signal.