System and / or method for speaker auto-calibration and speaker configuration layout estimation
Through the time-frequency masking technology based on noise-reliable time-frequency masking, the problem of automatic speaker calibration in low signal-to-noise ratio environment is solved, and the speaker position estimation and automatic calibration in background noise and reverberation is realized, improving the audio quality of immersive sounds .
Patent Information
- Application Number
- CN202410642933.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-23
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to achieve automatic speaker calibration in low signal to noise ratio environments, especially in the case of background noise and reverb, resulting in a degradation of audio quality generated by immersive sounds.
The arrival direction estimation method based on noise-rosted time-frequency masking is adopted, and the time-frequency masking block, feature frame detection block and generalized cross-correlation phase transformation block are used to extract the angle information between speakers through the feature tone signal and the inverse exponential sinusoidal scanning signal. Implement automatic speaker calibration and configuration layout estimation.
In low signal-to-noise ratio and reverberation environments, accurate speaker position estimation and automatic calibration are provided, improving the audio quality generated by immersive sound, capable of handling corner situations and background noise, ensuring robustness and accuracy of speaker arrays.
Smart Images

Figure CN119071697A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is generally related to U.S. application No. 18 / 204,165, with attorney case number P220103US (HARM0868PUS), filing date May 31, 2023, and title “BOUNDARY DISTANCE SYSTEM AND METHOD,” the disclosure of which is hereby incorporated by reference in its entirety.
[0003] This application is generally related to U.S. application No. 18 / 204,159, with agent case number P220104US (HARM0867PUS), filing date May 31, 2023, and title “APPARATUS, SYSTEM AND / OR METHOD FOR NOISE TIME-FREQUENCY MASKING BAS EDDIRECTION OF ARRIVAL ESTIMATION FOR LOUDSPEAK ER AUDIO CALIBRATION,” the disclosure of which is hereby incorporated by reference in its entirety. Technical Field
[0004] Various aspects disclosed herein generally relate to an apparatus, system and / or method for noise robust time-frequency masking based direction of arrival estimation for speaker audio calibration. These and other aspects will be discussed in more detail herein. Background Art
[0005] Various speaker manufacturers or providers may group various speaker categories together to form an ecosystem. In this regard, the various speakers communicate or work together with each other and / or with the mobile device. Thus, such speakers may achieve higher audio quality using immersive sound. Information related to the position of the speakers may be required for immersive sound generation. Therefore, automatic calibration may be required before the speakers can generate immersive sound. Summary of the invention
[0006] In at least one embodiment, an audio system is provided. The audio system includes a plurality of speakers and a mobile device. The plurality of speakers can be positioned in a listening environment and are arranged to transmit audio signals in the listening environment. Each speaker is programmed to determine angle information of an audio signal received at the speaker relative to other speakers in the plurality of speakers, and transmit a first signal indicating the angle information. The mobile device is programmed to receive the first signal from each of the speakers and determine the position of each speaker in the listening environment based at least on the angle information.
[0007] In at least another embodiment, a method is provided. The method includes transmitting an audio signal in a listening environment via a plurality of speakers that can be positioned in the listening environment, and determining, by each speaker, angle information of the audio signal received at each speaker relative to other speakers in the plurality of speakers and transmitting a first signal indicating the angle information. The method also includes receiving the first signal from each of the speakers at a mobile device and determining a position of each speaker in the listening environment based at least on the angle information.
[0008] In at least another embodiment, an audio system includes a plurality of speakers and a master speaker. The plurality of speakers are positionable in a listening environment and are arranged to transmit audio signals in the listening environment. Each speaker is programmed to determine angle information of an audio signal received at the speaker relative to other speakers in the plurality of speakers and transmit a first signal indicating the angle information. The master speaker is programmed to receive the first signal from each of the speakers and determine the position of each speaker in the listening environment based at least on the angle information. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Embodiments of the present disclosure are particularly pointed out in the appended claims. However, other features of the various embodiments will become more apparent and will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which:
[0010] Figure 1 Generally depicted is a system for performing noise robust time-frequency masking based directional and speaker automatic calibration and speaker configuration layout estimation according to one embodiment;
[0011] Figure 2 Describes how to remove noise by applying time-frequency (TF) masking according to one embodiment;
[0012] Figure 3 depicts the output of a TF masking block according to one embodiment;
[0013] Figure 4 Depicts two-microphone direction-of-arrival estimation using time-difference-of-arrival;
[0014] Figure 5 depicts various signature tone signals according to one embodiment;
[0015] Figure 6 Described is a method for performing optimized speaker auto-calibration and speaker configuration estimation according to one embodiment;
[0016] Figure 7 depicts an example of a speaker and microphone configuration according to one embodiment;
[0017] Figures 8 to 9 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and loudspeaker layout estimation operations of the method;
[0018] Figures 10 to 12 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 An example of outlier detection operation using the method;
[0019] Fig.13 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 An example of a reference speaker selection operation of the method;
[0020] Fig.14 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and loudspeaker layout estimation operations of the method;
[0021] Fig.15 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Examples of microphone orientation and loudspeaker layout estimation operations of the method;
[0022] Fig.16 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0023] Fig.17 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0024] Fig.18 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0025] Fig.19 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0026] Fig. 20 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of a candidate coordinate estimation operation of the method;
[0027] Fig.21 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 An example of the optimal coordination selection operation of the method;
[0028] Fig. 22 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Another example of the best coordination selection operation of the method;
[0029] Fig.23 depicts an example of microphone orientation and speaker estimation according to an embodiment;
[0030] Fig.24 A system for performing boundary estimation according to one embodiment is depicted;
[0031] Fig.25 depicts an example of a frequency response of an exponential sine sweep (ESS) for stimulating a listening environment according to one embodiment;
[0032] Fig.26 depicts an example of an amplitude spectrum of an inverse filter according to one embodiment;
[0033] Fig. 27 depicts an example of a room impulse response (RIR) measurement according to one embodiment;
[0034] Fig.28 depicts an example of peak detection involving RIR measurements according to one embodiment; and
[0035] Fig.29 A method for performing boundary estimation involving multiple speakers is described according to one embodiment. DETAILED DESCRIPTION
[0036] As required, detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to employ the present invention in various ways.
[0037] Noise-robust time-frequency masking based direction-of-arrival estimation for loudspeaker automatic calibration
[0038] The speaker is used to generate an immersive sound effect. For one aspect of immersive sound, it is necessary to perform automatic calibration to locate the position of the speaker. A method for performing speaker positioning includes estimating the azimuth of the speaker, which is also called the direction of arrival estimation (DOA). The performance of the DOA method may be problematic for low signal-to-noise ratios (SNR) (i.e., less than 0 dB) because noise is the dominant signal for low SNR conditions. In addition, in real scenarios, noise may not be avoided for the automatic calibration stage. Therefore, the automatic calibration stage requires a noise-robust DOA estimation method. The disclosed system and / or method utilizes a time-frequency (TF) mask that can be used for source separation as a preprocessing step of the DOA estimation method to achieve high performance at low SNR. The TF mask can extract the desired signal from the noisy signal captured by the microphone. The various aspects provided herein also provide a characteristic signal that maximizes performance under low SNR conditions. The embodiments disclosed herein provide a DOA estimation based on TF masking designed using at least two microphones and a characteristic signal that can be played back by a speaker.
[0039] As described above, for immersive sound generation of loudspeakers, automatic calibration is generally required. Failure in the automatic calibration phase may lead to negative feedback from the listener. In addition, since the environment cannot be controlled during the automatic calibration phase, background noise cannot be avoided. Therefore, for immersive sound generation using multiple loudspeakers, noise robust automatic calibration is desired. The disclosed embodiments provide noise robust automatic calibration to provide immersive sound generation. In addition, the disclosed system generally provides accurate DOA estimates for loudspeaker automatic calibration under low signal-to-noise ratios and reverberation. These aspects enable immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner situations (such as two loudspeakers, for example, on the same line but not aligned with each other).
[0040] One way of speaker auto-calibration may involve estimating the angle of the speaker, which is also referred to as DOA. There are many techniques for estimating the DOA of a talker / speaker, such as time difference of arrival (TDOA), multiple signal classification (MUSIC), and steered response power (SRP). While the TDOA method does not provide satisfactory performance for low signal-to-noise ratios (SNRs), MUSIC and SRP require a large number of microphones to achieve high performance at low SNRs. For low SNR conditions (i.e., -10 dB babble noise), even the performance of the MUSIC and SRP methods is not up to standard. The disclosed system provides a characteristic tone in the form of an inverse exponential sine sweep (ESS) signal, which has been found to provide an indication to the controller to initiate speaker auto-calibration, particularly in noisy environments such as -10 dB. In these types of noisy environments, the controller may not be able to perceive other types of characteristic tones that do not utilize ESS-based signals.
[0041] Figure 1 A system 100 for performing noise robust time-frequency masking based directional and speaker automatic calibration and speaker configuration layout estimation according to one embodiment is depicted. The system 100 includes a speaker 102 having a plurality of microphones 106a to 106b (or "106"), a time-frequency (TF) masking block 108, a feature frame detection block 110, a generalized cross-correlation (GCC) phase transform (PHAT) block 112, at least one controller 122 ("controller 122"), and a memory 130. The system 100 also includes a mobile device 150 having a matrix array 114, a microphone orientation estimation (MOE) block 116, an outlier detection block 118, an optimization block 120, and at least one controller 123 (hereinafter referred to as "controller 123"). It should be appreciated that the controller 123 may execute any instructions for any function performed by the mobile device 150 as described herein. Although Figure 1 A single speaker 102 is shown, but it should be appreciated that the system 100 includes any number of speakers 102 positioned therein.
[0042] At least one of the speakers 102 transmits an audio signal including the characteristic tone 104 into the listening environment 151 to the other speakers 102 in the system 100. It should be appreciated that the speakers 102 typically include at least two of the microphones 106a to 106b. The speakers 102 may transmit an audio signal including the characteristic tone 104 into the listening environment 151. The microphones 106a to 106b positioned on the different speakers 102 capture the audio signal including the characteristic tone 104. Each speaker 102a and 102b includes a memory 130. The memory 130 of the speaker 102b stores the audio signal and the corresponding characteristic tone (or characteristic frame) 104 for processing.
[0043] As described above, the TF masking block 108, the feature frame detection 110, the GCC PHAT block 112, and the controller 122 are implemented in all speakers 102 present in the system 100. For example, assume that the system 100 includes four speakers 102, and the first speaker 102 receives audio signals and corresponding feature tones 104 from the other speakers 102. Therefore, in this regard, each speaker 102 estimates the direction of arrival (DOA) of the audio signals received from the other three speakers 102. The mobile device 150 includes one or more transceivers 155 to wirelessly receive the DOA estimates from each of the speakers 102 in the system 100. It is also recognized that each of the speakers 102 in the system 100 may also include one or more transceivers 152 to wirelessly transmit the estimated DOA information to the mobile device 150.
[0044] In general, the TF masking block 108 in the speaker 102 reduces the noise effect associated with the captured audio signal received from other speakers 102 in the system 100. For example, the controller 122 applies the TF masking block 108 to each microphone input to reduce the noise effect. After the TF masking block 108 reduces the noise effect, the feature frame detection block 110 estimates the characteristic tone 104. In one example, the length of the characteristic tone 104 can be 200 milliseconds. However, the speaker 102 records the received audio, for example, for more than 200 milliseconds, because the speaker 102 does not know when another speaker 102 plays the characteristic tone 104. It can be assumed that the speaker 102 may be in a recording mode while the other speaker 102 transmits the characteristic tone 104. It is generally desirable to detect the characteristic tone 104 for a sufficiently long duration to correctly estimate the DOA. Receiving the characteristic tone 104 on the audio signal can indicate to the receiving speaker 102 that the system 100 may be in an automatic calibration mode. In the automatic calibration mode, the speakers 102 may transmit information corresponding to the positions of the speakers 102 relative to the mobile device 150 (or other audio source).
[0045] The controller 122 applies a cross-correlation between the characteristic tone 104 played by the transmitting speaker 102 and the acquired audio. The cross-correlation performed by the GCC PHAT block 112 provides the location of the characteristic tone 104 in the long recording. In this regard, the controller 122 uses the location to extract the characteristic tone 104. At this point, the extracted characteristic tone 104 is provided to the GCC-PHAT block 112. The controller 122 can then use the estimated DOA to perform automatic calibration of the speaker 102b. These aspects will be discussed in more detail below. Returning to the TF masking block 108, the controller 122 applies a TF masking operation as a preprocessing step for DOA estimation. The TF masking block 108 can eliminate the TF frequency band where the noise is most dominant in the audio signal to minimize the effects of noise and reverberation. Combined with Figure 2 A noisy input audio signal including a characteristic tone 104 is generally shown at 200. Figure 2 The noisy input audio signal shown, including the characteristic tone 104, comprises a noisy swept sine wave between 6 kHz and 7 kHz. Figure 2 In response to the TF masking block 108 performing a TF masking operation, the controller 122 extracts a characteristic tone 104 or signal from the noise mixture of the input audio signal, as generally shown at 202 in FIG.
[0046] Return to reference Figure 1 , TF masking block 108 uses techniques for source separation and speech enhancement. TF masking block 108 receives characteristic tone 104 to generate an enhancement signal. Controller 122 generates a sampling delay signal using characteristic tone 104 via the enhancement signal The controller 122 uses the sampling delay To determine the DOA of the audio signal received at the receiving speaker 102. The TF masking-based techniques described above may include ideal binary mask (IBM), ideal ratio mask (IRM), complex ideal ratio mask (cIRM), optimal ratio mask (ORM), etc. In general, when TF-based DOA estimation is adopted, such masking techniques should not modify the phase information. Based on various requirements and tests, the disclosed system 100 may adopt the IRM defined by:
[0047]
[0048] References to Equation 1 may be found, for example, in "The Optimal Ratio Time-Frequency Mask for Speech Separation in Terms of Signal-to-Noise Ratio," The Journal of the Acoustical Society of America 134, No. 5 (2013): EL452-EL458. When S(t, f) is the frequency response of the characteristic signal (or characteristic tone 104), N(t, f) represents the noise spectrum and β is a smoothing factor. Due to the overall knowledge of the characteristic tone 104, S(t, f) can be calculated. The denominator in equation (1) can be the captured signal at the microphones 106a to 106b. After the controller 122 calculates the mask, the enhanced signal can be calculated using the product of the captured signal and the mask, as shown in equation (2).
[0049] E(t, f)=IRM(t, f)*Y(t, f) (2)
[0050] E(t, f) represents an enhanced signal, which is a two-channel signal given that the two microphones 106a and 106b of the receiving speaker 102 each receive an incoming audio signal including the characteristic tone 104. Y(t, f) corresponds to the captured signal at the microphones 106a-106b. The enhanced signal may correspond to Figure 2 0044] A signal is generally shown at 202 in which noise is removed from a captured audio signal.
[0051] Figure 3 An example of a recorded signal 300 provided by a TF masking block 108 of a speaker 102 is depicted according to one embodiment. For example, the recorded signal 300 corresponds to the output provided by the TF masking block 108 after the TF masking block 108 performs a masking operation. In general, the recorded signal 300 provided by the TF masking block 108 may include a long string of audio data 302 that includes silence / noise and characteristic tones 104. The characteristic tones 104 are typically delimited by frames 304. As shown in FIG. Figure 3The two audio signals shown correspond to one audio signal received at microphone 106a and another audio signal received at microphone 106b, which are then processed by TF masking block 108. Controller 122 detects frame 304 using cross-correlation between audio data 302 and characteristic tones 104. In general, controller 122 detects an enhanced version of frame 304 (e.g., characteristic tones 104 and acquired audio signals (e.g., which correspond to recorded signal 300, audio data 302, and frame 304)) to detect the start time of frame 304. GCC PHAT block 112 then receives frame 304 as input.
[0052] refer to Figure 1 and Figure 4 , the GCC PHAT block 112 processes the output of the feature frame detection block 110 to provide an estimated DOA of the captured audio signal transmitted by at least the speaker 102. An example of the operation of the GCC PHAT is described in "The Generalized Correlation Method for Estimation of Time Delay", IEEE Transactions on Acoustics, Speech, and Signal Processing 24, No. 4 (1976): 320-327, which is incorporated herein by reference in its entirety. As described above, the speaker 102 in the system 100 can provide (or wirelessly transmit) the estimated DOA reading to the mobile device 150 (see Figure 1 ). The mobile device 150 may be a cell phone, a laptop computer, a desktop computer, etc. Also as described above, the speakers 102 in the system 100 may include one or more transceivers 152 to wirelessly transmit and receive information (including estimated DOA readings) to each other and / or to the mobile device 150.
[0053] The GCC PHAT block 112 may utilize single path wave propagation of sound waves from a single sound source signal s(n) provided by a sound source (or any of the speakers 102). Microphones 106a and 106b receive signal s(n) as received signal x 1 (n) and x 2 (n), which are delayed and attenuated versions of the original sound signal s(n). In general, the controller 122 can be obtained by finding x 1 (n) and x 2 The received signal x is determined by the maximum value of the cross-correlation of 1 (n) and x 2 (n) The controller 112 performs the cross-correlation by executing the following equation:
[0054]
[0055]
[0056]
[0057]
[0058]
[0059] Equations 3 to 7 are used in the GCC PHAT block 112 to estimate the sampling delay Equation 3 represents x 1 (n) and x 2 Equation 4 is the cross power density, which is obtained by taking x 1 (n) and x 2 The PHAT processor (of the GCC PHAT block 112) is shown in Equation 5. Applying the inverse Fourier transform yields x as shown in Equation 6. 1 (n) and x 2 Finally, by finding x in Equation 7 1 (n) and x 2 (n) to calculate the sampling delay
[0060] At this time, the controller 122 may determine the DOA of the received audio signal or the angle of the sound source 102a (or the first speaker 102a). For example, the controller 122 may determine the DOA (or angle information, “angle”) of the audio signal received by the receiving speaker 102 by:
[0061]
[0062] in is an estimate of the sampling delay as described above, c is the speed of sound, and d is the distance between microphones 106a and 106b, which is a known value. GCC Phat block 112 estimates the phase difference between the audio captured between microphones 106a and 106b. Thus, the phase difference generally corresponds to as described in Equation 8 (or angle information). The controller 122 converts the phase difference into an enable as described in Equation 8, in particular using arc cosine. The controller 122 determines the sampling delay The way in Figure 4 Shown in.
[0063] Figure 5Various characteristic tone signals 500, 502 are depicted according to one embodiment. In general, the characteristic tone 500 includes energy below 4kHz. The characteristic tone 500 can be generated based on an exponential sine sweep (ESS). In this case, for a higher noise based environment, it may be more desirable to provide a characteristic signal that includes more energy at high frequencies to perform the estimated DOA. The characteristic tone signal 502 is generated based on an inverse ESS. Figure 5 It is shown that both the characteristic tone signals 500, 502 are in the frequency domain. The characteristic tone signal 502 has a higher amplitude after 1kHz, which prevents the characteristic tone signal 502 from being distorted by background noise. In one example, the characteristic tone 104 generated by the first speaker 102a or the second speaker 102b may be based on an inverse ESS from a predetermined frequency range, which may be 700Hz to 10kHz and has a predetermined length of, for example, 150ms at a predetermined frequency of, for example, 48kHz. The disclosed system 100 generally provides accurate DOA estimation for speaker automatic calibration under low signal-to-noise ratios and reverberation. These aspects enable immersive sound generation and microphone array calibration. In addition, the disclosed system can accurately estimate DOA for corner situations (such as two speakers, for example, in the same line but not aligned with each other). It should be appreciated that the characteristic tone signals 500, 502 (e.g., inverse EES signals) have been found to tolerate high noise environments at least at a level of -10dB. For example, it has been found that the inverse ESS signal is not affected in high noise environments of at least −10 dB, which is used to provide a sufficient signal to trigger automatic calibration of the various speakers 102 in the system 100 and determination of DOA.
[0064] Optimization of automatic speaker calibration and speaker configuration layout estimation
[0065] As described above, the speakers 102 in the system 100 are configured to communicate with each other. Each of the first speaker 102a and the second speaker 102b can provide high audio quality while utilizing immersive sound. The immersive sound technology depends on the positions of the first speaker 102a and the second speaker 102b. Therefore, in this regard, the immersive sound technology requires an automatic speaker calibration process.
[0066] There are multiple ways to perform automatic calibration. One way to perform automatic calibration requires providing an estimate of the speaker azimuth, also referred to as DOA. The DOA of the audio signal transmitted from each speaker can be detected by playing a characteristic tone from one speaker at a time. Then, the angles (or DOAs) of different speakers are used to create the speaker configuration in the room. In some cases, due to environmental conditions and the position of the speakers, the estimate of the azimuth obtained may be wrong. Such errors may mainly occur when the speakers are not aligned with each other (for example, due to speaker directivity) and the background noise has more energy than the characteristic tone. Since these aspects may occur in real-world scenarios, the automatic calibration technology implemented in the speaker should handle these scenarios. The system 100 disclosed herein provides multiple DOA estimates for optimizing speaker positions and estimating speaker layout configurations of two or more speakers. The system 100 also provides an accurate representation of the speaker configuration required for a true immersive experience. The disclosed embodiments can improve robustness and overcome the above-mentioned environmental conditions. In addition, the disclosed embodiments can provide (i) accurate speaker configuration estimation, (ii) speaker orientation estimation, (iii) detection of DOA estimate outliers when considering background noise, reverberation, and obstacles, and (iv) optimizing speaker configuration estimation based on previous DOA estimation and outlier detection.
[0067] Return to reference Figure 1 , the system 100 also includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. The matrix block 114 stores the DOA estimates of each of the first speaker 102a and the second speaker 102b. As described above, it should be appreciated that the system 100 may include any number of speakers 102, but preferably, more than two speakers 102 may be required. The microphone orientation estimation block 116 estimates the orientation of each of the microphones 106a and 106b positioned on the speaker 102. The outlier detection block 118 detects outliers that may exist in the matrix formed by the matrix block 114. These outliers or errors may be attributed to erroneous DOA estimates or obstacles between the first speaker 102a and the second speaker 102b. The optimization block 120 performs reference microphone selection, initial layout estimation, candidate coordinate estimation, and optimal coordinate selection. These aspects will be discussed in more detail below.
[0068] Figure 6 Depicted is a method 600 for performing optimized speaker auto-calibration and speaker configuration estimation according to one embodiment.
[0069] In operation 602, the microphone orientation estimation block 116 estimates the orientations of the microphones 106a and 106b. Figure 8 and Fig. 9 This operation is discussed in more detail.
[0070] In operation 604, the outlier detection block 118 detects possible outliers in the matrix formed by the matrix block 114 with respect to the DOA. Figure 10 to Figure 11 This operation is discussed in more detail.
[0071] In operation 606, the optimization block 120 performs reference microphone selection. Fig.12 This operation is discussed in more detail.
[0072] In operation 608, the optimization block 120 uses the DOA estimate to perform an initial layout estimate. Fig.13 This operation is discussed in more detail.
[0073] In operation 610, the optimization block 120 calculates candidate coordinate estimates. Figures 14 to 16 This operation is discussed in more detail.
[0074] In operation 612, the optimization block 120 selects the best coordinates. Figure 17 to Figure 18 This operation is discussed in more detail.
[0075] Figure 7 An example of a speaker and microphone configuration 700 in system 100 according to one embodiment is depicted. Configuration 700 includes Figure 1 The speaker 102. Reference Figure 7 and below, Figure 1 The speakers 102 are generally shown as a first speaker 102a, a second speaker 102b, a third speaker 102c and a fourth speaker 102d. Figure 1As described above, any number of speakers may be provided. Each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a first microphone 106a and a second microphone 106b. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d includes a controller 122, a memory 130, and a transceiver 152. Similarly, each of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 120d includes a TF masking block 108, a feature frame detection block 110, and a GCC PHAT block 112. Also as described above, the mobile device 150 includes a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. It should also be recognized that in another embodiment, the system 100 may include a main speaker 103. The primary speaker 103 may correspond to any of the speakers 102a to 102d and may simply be designated as the primary speaker to perform similar tasks as the mobile device 150. For example, the primary speaker 103 may be arranged to provide a layout of the speakers 102, including the layout of the primary speaker 103, based on the principles disclosed herein. In this sense, if it may be preferable for the primary speaker 103 to provide the positions of the various speakers 102 and 103 within the listening environment 151 for the purpose of establishing channel assignments for the speakers 102 and 103, the primary speaker 103 provides a similar level of functionality as that provided in conjunction with the mobile device 150. Thus, in this regard, the primary speaker 103 may include a matrix block 114, a microphone orientation estimation block 116, an outlier detection block 118, and an optimization block 120. While the primary speaker 103 may provide the location of the speakers 102 , 103 in the listening environment 151 in a manner similar to that explained with the mobile device 150 , the primary speaker 103 may not provide any visual indicators or cues to the user regarding the location of the speakers 102 , 103 .
[0076] The first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d wirelessly communicate with each other and / or with the mobile device 150 via the transceiver 152 to provide a layout of speakers in the listening environment 151. In particular, the mobile device 150 can provide a layout of the various speakers 102a, 102b, 102c, and 102d arranged in the listening environment 151. In general, the specific layout of the speakers 102a-102d relative to each other may be unknown, and the various aspects set forth herein can determine the specific layout of the speakers 102a-102d in the listening environment 151. Once the layout of the speakers 102a-102d is known, the mobile device 150 can assign channels to the speakers 102a-102d in a determined manner based on a pre-stored or predetermined system configuration.
[0077] The mobile device 150 may display the layout of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d based on the information received from these devices. In one example, the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d may wirelessly transmit DOA estimates, microphone orientation estimation information, outlier information, reference speaker selection information, initial speaker layout estimation, candidate coordinate estimation information, and optimal coordinate selection information to each other via the transceiver 152 and / or to the mobile device 150, as described in method 600.
[0078] A legend 702 is provided that illustrates various angles of the position of microphones 106a-106b on one speaker 102 relative to the positions of microphones 106a-106b on other speakers 102a, 102b, 102c, and 102d. Various operations of the method 600 will be described below with reference to the legend 702. The first speaker 102a, the third speaker 102c, and the fourth speaker 102d illustrate their respective microphones 106a-106b arranged horizontally on such speakers 102a, 102c, and 102d. The second speaker 102b illustrates the microphones 106a-106b arranged vertically on the second speaker 102b. It should be appreciated that the arrangement of the microphones 106a-106b is unknown before the speaker layout is determined, and the arrangement of the microphones 106a-106b may be arranged in any number of configurations on the speakers 102a-102d in the listening environment 151. The disclosed system 100 and method 600 are configured to determine a speaker configuration layout while taking into account different configurations of the microphones 106a - 106b .
[0079] Referring to the first speaker 102a and further referring to the legend 702, the first speaker 102a captures audio (or detects audio) from the second speaker 102b at 0 degrees. The first speaker 102a captures audio (or detects audio) from the third speaker 102c at 45 degrees. The first speaker 102a captures audio from the fourth speaker 102d at a 90 degree angle. Figure 7The angles (or angle information) at which the remaining speakers 102b to 102d receive audio relative to the other speakers 102a to 102d are shown. Any reference to the term "angle" may also correspond to "angle information" and vice versa. The relevance of the angles (or angle information) will be discussed in more detail below. It should be appreciated that each of the speakers 102a to 102d transmits information related to the angle information at which they receive audio from each other to the mobile device 150 or other suitable computing device. The mobile device 150 stores the angles in its memory. The DOA information reported by the speakers 102a to 102d is reported as the angles mentioned above.
[0080] Figures 8 to 9 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 An example of microphone orientation and operation 604 of method 600 of . At least one embodiment provides a dual speaker position in space that establishes a line, and the slope of the line does not change when viewed from one speaker or another. In general, the system 100 and / or method 600 recognizes that the dual speaker position in space establishes a line, and the slope of the line does not change from one speaker to another. A first matrix 800 is shown, which depicts the relative angles of the audio received relative to various speakers 102a to 102d (or S1 to S4, respectively). Any angle reading of -360 represents a null value. For example, the first matrix 800 shows that S1 in both the columns and rows of the matrix is -360 because the first speaker 102a (or S1) cannot receive audio from itself. Any value of the angle shown as 360 for the second speaker 102b (or S2), the third speaker 102c (or S3) and the fourth speaker 102d (or S4) further illustrates this point.
[0081] The mobile device 150 generally stores information corresponding to the angle information depicted in the first matrix 800. The first column, as shown by the dashed box shown in the first matrix 800, corresponds to a particular speaker that is receiving audio from the speakers S1 to S4 shown in columns 2 to 5, respectively. For example, referring to the first column and the second row, the second speaker (e.g., or S2) 102b receives audio from the first speaker (e.g., or S1) 102a (as shown in the second column) at a 90-degree angle, the second speaker 102b receives audio from the third speaker 102c at a 0-degree angle, and the second speaker 102b receives audio from the fourth speaker 102d (or S4) at a 45-degree angle. Referring to the first column and the third row, the third speaker 102c (e.g., or S3) receives audio from the first speaker 102a (e.g., or S1) at a 45-degree angle, and the third speaker 102c (e.g., or S3) receives audio from the fourth speaker 102d (e.g., or S4) at a 0-degree angle. Referring to the first column and the fourth row, the fourth speaker 102d receives audio from the first speaker 102a (or S1) at a 90 degree angle, the fourth speaker 102d receives audio from the second speaker 102b (or S2) at a 135 degree angle, and the fourth speaker 102d receives audio from the third speaker 102c (or S3) at a 180 degree angle.
[0082] See also Fig. 9 , the mobile device 150 receives information corresponding to various angles from the transceivers 152 of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. As described above, the mobile device 150 assembles the first matrix 800 based on the information received from the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d, respectively. The mobile device 150 may determine the orientations of the microphones 106a to 106b of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to each other. In particular, the mobile device 150 may determine whether the orientations of the microphones 106a to 106b of the various first speakers 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are different from each other based on the angles stored in the first matrix 800. The embodiments disclosed herein generally illustrate that the slope may not change, but the angle depends on the orientation of the microphones 106a - 106b , which may be from the first row and first column generally shown at 900 .
[0083] For example, the mobile device 150 may determine whether the angle differences between the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d shown in the first matrix 800 correspond to one or more predetermined values (e.g., 0 or 180). If the differences between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d correspond to one or more predetermined values, the mobile device 150 may determine that the microphones 106a to 106b of the two or more speakers 102a, 102b, 102c, 102d are in the same orientation. If the differences between the angles of the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d do not correspond to one or more predetermined values, the mobile device 150 may determine that the microphones 106a to 106b of the two or more speakers 102a, 102b, 102c, 102d are not in the same orientation.
[0084] refer to Fig. 9 In the first matrix 800 shown, the mobile device 150 determines that the second speaker 102b has a 90 degree difference relative to the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. Therefore, in this regard, the mobile device 150 determines that the orientation of the microphones 106a-106b of the second speaker 102b is different from the orientation of the microphones 106a-106b of the first speaker 102a, the third speaker 102c, and the fourth speaker 102d. This Fig. 9 Shown in.
[0085] In general, the mobile device 150 subtracts the angle of the first column from the first row to perform microphone orientation estimation. Then, a subtraction operation is performed, and the result is [0, 90, 0, 0] for the first speaker 102a (or S1), the second speaker 102b (or S2), the third speaker 102c (or S3), and the fourth speaker 102d (or S4). Therefore, the microphone estimate of the third speaker 102c (S3) and the fourth speaker 102d (S4) is 0, which is the same as the orientation of the first speaker 102. The mobile device 150 can also perform microphone orientation using a modulo operation after performing the subtraction operation, because the angle range should be [0, 180], such as Figure 7 702 of FIG. In general, the slope between the speakers 102a to 102d may be constant, but the angle may depend on the orientation of the microphones 106a to 106b. The mobile device 150 generates a microphone orientation array 900 that includes an angle difference that does not correspond to 0 or 180 degrees (or a predetermined value). The embodiments disclosed herein generally show that the slope may be constant, but the angle depends on the orientation of the microphones 106a to 106b, which can be found from the first row and the first column of the microphone orientation array.
[0086] Figures 10 to 12 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 6. After performing the microphone orientation estimation of operation 602, the mobile device 150 performs the outlier detection operation 604 to determine whether any of the speakers 102a to 102d is an outlier relative to the layout. If any of the speakers 102a to 102d is determined to be an outlier, the mobile device 150 determines that the position of the speaker 102 relative to the positions of the other speakers 102a to 102d is incorrect or cannot be determined.
[0087] refer to Fig.10 As part of operation 604, the mobile device 150 subtracts the microphone orientation array 900 from the first matrix 800 to provide a calibrated angle matrix 902. The mobile device 150 takes into account speakers whose microphone orientations are not aligned with the rest of the microphones in the speaker. Fig.10 The calibrated angle matrix 902 is shown, showing the 90 degree angles from the microphone orientation array 900 subtracted from the angles in the second row (S2) (except -360 as this is a null value) to provide the following in the calibrated angle matrix 902 [0, -360, 90, 135].
[0088] refer to Fig.11 According to one example, as part of operation 604, the mobile device 150 may compare the angles shown in row 2 of the calibrated angle matrix 902 to a predetermined threshold. If any one or more angles in row 2 of the calibrated angle matrix 902 are above the predetermined threshold, the mobile device 150 detects an outlier of one or more speakers 102a to 102d having an angle above the predetermined threshold. The mobile device 150 generates a block matrix 1100, such as Fig.11 As shown generally. For example, the mobile device 150 checks the difference between each pair of estimates (e.g., the angle estimate of the second speaker 102b at the first speaker 102a and the angle estimate of the first speaker 102a at the second speaker 102b). For example, the mobile device 150 may apply modulo 180 to ensure that the difference is within the range of [0,180] degrees. If the difference is above a predetermined threshold, the mobile device 150 may determine that an outlier exists for the pair of speakers. In this example, the block matrix 1100 does not indicate an error in any of the speakers 102a to 102d. Therefore, in this regard, the angle in row 2 of the calibrated angle matrix 902 is less than the predetermined threshold. Outliers generally represent various erroneous estimates in the DOA matrix. Detected outliers may not be used for optimization (e.g., operations 606, 608, and 610).
[0089] refer to Fig.12 And similar to combining Fig.11 According to another example, as part of operation 604, mobile device 150 may compare the angle shown in row 2 of calibrated angle matrix 902 with a predetermined threshold. Fig.12 In the example shown, row 2 of the calibrated angle matrix 902 is different from Fig.11 , 360°, 25°, 135°. In this regard, when the mobile device 150 compares the angle shown in row 2 of the calibrated angle matrix 902 to a predetermined threshold, the mobile device 150 generates a value of "1", as shown in row 3, column 2 and row 2, column 3 in the block matrix 1100. Referring to the block matrix 1100, all values of 0 do not represent outliers, and as described above, an angle of -360° corresponds only to a null value and can be ignored. In this case of the block matrix 1100, the third speaker 102c is determined to be an outlier relative to the second speaker 102b. The estimates in row 2, column 3 and row 3, column 2 will not be used in operations 606, 608, 610.
[0090] Fig.13 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 606 of the reference speaker selection operation of method 600. The mobile device 150 may then check any rows / columns of the block matrix 1100 that are filled with "1". As described above, these values generally indicate that the speaker is an outlier. Fig.13 The block matrix 1100 shown is similar to the combination of Fig.11 The blocking matrix 1100 is shown and does not indicate the presence of any outliers. If the mobile device 150 does not detect an outlier in the blocking matrix 1100, the mobile device 150 generates an error and repeats the method 600 again.
[0091] Fig.14 and Fig.15 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 An example of microphone orientation and speaker layout estimation operation 608 of method 600. Fig.14, the configuration of the first, second, third and fourth speakers 102a to 102d shown is now reflected as including distance coordinates in the x and y axes. As shown, the first speaker 102a is selected as the reference speaker. The second speaker 102b has coordinates (100, 0) relative to the first speaker 102a, the third speaker 102c has coordinates (70.71, -70.71) relative to the first speaker 102a, and the fourth speaker 102d has coordinates (0, -100) relative to the first speaker 102a. As described above, the mobile device 150 does not know the exact layout of the speakers 102a to 102d in the listening environment 151. In this way, the mobile device 150 establishes a reference matrix 1400 with reference coordinates (or distances or values): 0, 100, 100, 100 for the first speaker 102a, the second speaker 102b, the third speaker 102c and the fourth speaker 102d, respectively. The mobile device 150 randomly selects coordinates (e.g., 0, 100, 100, 100). In this case, the mobile device 150 assumes that the second speaker 102b, the third speaker 102c, and the fourth speaker 102d are located equally far from the first speaker 102a. As shown in the first matrix 800, the mobile device 150 has information corresponding to the angles relative to the audio received by the first speaker 102a, the second speaker 102b, the third speaker 102c, and the fourth speaker 102d. However, the actual distances of such speakers 102a-102d are unknown.
[0092] Reference now Fig.15 , the mobile device 150 calculates the distances (or x, y coordinates) of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a, which can be determined based on the following equation 9:
[0093] (x,y)(x s1 +Distance s1sj *cos(DOA s1sj ), y s1 -distance s1sj *sin(DOA s1sj )
[0094] For example, the mobile device 150 may calculate the distance coordinates of the second speaker 102b, the third speaker 102c, and the fourth speaker 102d relative to the first speaker 102a based on equations 10, 11, and 12, respectively:
[0095] (x 2 ,y 2 )=(100*cos(0),-100*sin(0))=(100,0)
[0096] (x3 ,y 3 )=(100*coS(45),-100*sin(45))=(70.71,-70.71)
[0097] (x 4 ,y 4 )=(100*cos(90),-100*sin(90))=(0,-100)
[0098] Equation 10, as shown above, corresponds to the distance coordinate of the second speaker 102b relative to the first speaker 102a, where the angle 0 is inserted into Equation 5 and taken from the first row (i.e., S1) and the second column (i.e., S2) of the first matrix 800. Equation 11, as shown above, corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle 45 is inserted into Equation 8 and taken from the first row (i.e., S1) and the second column (i.e., S3) of the first matrix 800. Equation 11, as shown above, corresponds to the distance coordinate of the third speaker 102c relative to the first speaker 102a, where the angle 90 is inserted into Equation 12 and taken from the first row (i.e., S1) and the third column (i.e., S3) of the first matrix 800.
[0099] Figures 16 to 20 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 Various aspects of the candidate coordinate estimation operation 610 of method 600 are described below. Fig.16 Generally, it is shown that based on the performance of operation 608, the estimate of the layout of the third speaker 102c is positioned at the coordinates (70.71, -70.71) relative to the first speaker 102a. However, the actual layout indicates that the third speaker 102c is positioned at the coordinates (70.71, -70.71) relative to the first speaker 102a, the third speaker 102c is positioned at the coordinates (100, -100) relative to the second speaker 102b, and the third speaker 102b is positioned at the coordinates (-100, -100) relative to the fourth speaker 102d. These aspects are generally shown as candidate coordinate estimates 1600. Fig.17 Shown based on the combination Fig.15The discussed equations 10, 11, and 12 determine the manner in which the various coordinates of the third speaker 102c are determined relative to the first, second, and fourth speakers 102a to 102d. In general, the mobile device 150 does not know whether the coordinates of the third speaker 102c are correct. In this case, the mobile device 150 estimates possible candidate points. In operation 612, the mobile device 150 calculates the error for each candidate point. The candidate that exhibits the lowest error is selected as the best coordinate. It should be recognized that all DOA estimates from all speakers 102 are transmitted to the mobile device 150 using any number of wireless communication protocols (such as but not limited to Bluetooth, WiFi, etc.). In Fig.17 In the example shown, the mobile device 150 utilizes angles from the calibrated angle matrix 902 to determine the coordinates of the third speaker 102c relative to the first speaker 102a, the second speaker 102b, and the fourth speaker 102d.
[0100] refer to Fig.18 , the mobile device 150 expands the candidate coordinate estimate 1600 by combining the x and y points. As generally shown at 1800, the candidate coordinate estimate 1600 is provided in addition to the expanded candidate coordinate estimate 1802. In general, the candidate coordinate estimate reflects the x and y coordinates in the following manner: for the first speaker 102a, the second speaker 102b, and the fourth speaker 102d, respectively (x a ,y a )、(x b ,y b ), and (x c ,y c The expanded candidate coordinate estimates 1802 reflect the x and y coordinates in the following manner: for the first speaker 102a and the second speaker 102b and the fourth speaker 102d and the first speaker 102a, respectively: (x a ,y b )、(x c ,y z). The expanded candidate coordinate estimate 1802 is expanded in the manner shown at 1800 because some degree estimates provide information in one dimension, for example (e.g., x coordinate, y coordinate). The mobile device 150 combines the coordinates to obtain information in a two-dimensional format (e.g., x coordinate and y coordinate). The mobile device 150 typically combines the candidate coordinate estimate 1600, the expanded candidate coordinate estimate 1802, and the expanded angle as shown at 1804. The expanded angle 1804 is typically estimated using the angles of the first speaker 102a and the second speaker 102b (e.g., S2 and S1) corresponding to (100, -100). The first speaker 102a and the third speaker 102c form a line, and the coordinates (100, -100) are calculated using the intersection of the two lines. The angle information is used to form the line. The mobile device 150 expands the candidate coordinate estimate to locate the third speaker 102c to the intersection between the second speaker 102b and the fourth speaker 102d. The mobile device 150 does not consider the candidate coordinate estimate 1600, the expanded candidate coordinate estimate 1802, and the expanded angle 1804 of the third speaker 102c because the position of the third speaker 102c is incorrect. In general, the mobile device 150 does not need to determine whether any position is correct. The figures described herein are provided only as examples. Fig.18 The operations for the third speaker 102 c are disclosed as an example. Throughout the method, these operations are applicable to each speaker 102 in the system 100 .
[0101] refer to Fig.19 , the mobile device 150 continues to expand the candidate coordinate estimate 1600. In this case, a combination of Fig.19 The example shown is to simply illustrate another example of the expanded candidate coordinates, and combined with Fig.19 The examples shown may be independent of the examples shown above.
[0102] Fig. 20 Another example of a candidate coordinate estimation operation 610 contemplated by mobile device 150 is depicted. Fig. 20Another example of a modified first matrix 800' and a modified blocking matrix 1100' is shown. The modified first matrix 800' shows that there is an obstacle between the fourth speaker 102d and the third speaker 102c, as shown by the angle "145" in the third column (e.g., S3) and the fourth row (e.g., S4). In general, for a four-speaker layout configuration, the method 600 can only tolerate a single outlier between any two speakers. In this case, the mobile device 150 determines two or more outliers (e.g., the third speaker 102c and the fourth speaker 102d). As discussed above in conjunction with operation 604, the mobile device 150 determines that the third speaker 102c is an outlier. In this regard, the mobile device 150 also determines that the fourth speaker 102d is an outlier based on the description provided above in conjunction with operation 604. In view of the presence of more than one outlier, the mobile device 150 includes a user interface and commands the user to move any obstacles that exist relative to the third speaker 102c and the fourth speaker 102d. Given that the third speaker 102c and the fourth speaker 102d are outliers (i.e., an obstacle is formed therebetween), the mobile device 150 does not consider the candidate coordinate estimates 1600 (or vice versa), the expanded candidate coordinate estimates 1802, and the estimates of the expanded angles 1804 from the third speaker 102c and the fourth speaker 102d because there is an outlier between the positions of the third speaker 102c and the fourth speaker 102d, and such estimates are considered incorrect. The mobile device 150 updates the modified blocking matrix 1100', which illustrates that the third speaker 102c and the fourth speaker 102d are blocked when considering the layout.
[0103] Fig.21 and Fig. 22 Depicted is a method according to one embodiment of the present invention. Figure 7 Executed on the configuration Figure 6 The mobile device 150 performs the optimal coordinate selection operation 612. For example, the mobile device 150 performs the following equation (13):
[0104]
[0105] in is the angle calculated using the candidate x and y coordinates of the ith speaker, and C corresponds to the index of the candidate. The mobile device 150 selects the candidate point that minimizes the error. The calibrated DOA matrix 800 set forth above is used as the DOA matrix in the above equation. ij .
[0106] Fig. 22 An example of the manner in which equation 13 is implemented is generally shown. It should be appreciated that if Fig. 22 shown It can be obtained by the following equation:
[0107]
[0108] The mobile device 150 determines the errors of the third speaker 102c and the first speaker 102a respectively based on Equation 13:
[0109] Error 31 = |45-45| + |90-150| + |0-20| = 80
[0110] Similarly, the mobile device 150 also determines the errors of the third speaker 102c and the second speaker 102b respectively based on Equation 10:
[0111] Error 32 = |45-45| + |0-0| + |180-180| = 0
[0112] Fig. 22 shows the use of Equation 10 for Fig. 22 The first two rows of the table are used to estimate the position of the third speaker. From the second term in Equation 5. Fig. 22 As shown in the table represents the angle between the speaker 102 and the candidate point.
[0113] Fig.23 An example of a microphone orientation and loudspeaker estimation method 600 is depicted according to one embodiment. Fig.23 is the position of the speakers 102a to 102d in the listening room 151 (or the listening environment). The mobile device 150 may display the positions (e.g., front, left, right, and rear) of the speakers 102a to 102d in the listening room. In general, the system 100 and / or the method 600 are based on at least Figures 6 to 22 The method shown determines the positions of the first, second, third, and fourth speakers 102a to 102d in the listening environment 151. The system 100 and / or method 600 utilizes the position information to provide channel allocation for immersive sound generation relative to the speakers 102a to 102d. For example, the mobile device 150 utilizes the final DOA to allocate the speakers 102a to 102d as front, left, right, and rear speaker positions.
[0114] System and method for boundary distance estimation
[0115] As shown above, the first, second, third, and fourth speakers 102a-102d are typically from a family of products, all of which are equipped with microphones 106a-106b mounted thereon. The microphones 106a-106b of each speaker 102 provide the ability to detect the location of an audio source (e.g., a mobile device 150) relative to any nearby walls. However, since the microphones 106a-106b may be arranged linearly when packaged on the corresponding speaker 102, the microphones 106a-106b may lack the ability to distinguish between audio sources in front of or behind the speaker based on using the line between the microphones 106a-106b as a line of symmetry. Detecting a wall or obstacle in one direction can eliminate the symmetry limitation.
[0116] In addition, if the speaker is placed too close to a wall or a corner, the speaker may not be detected. The disclosed system can detect whether the speaker is placed too close to the wall, and automatically correct the speaker positioned close to the wall to ensure that the desired sound field is transmitted in the room (or listening environment 151). In general, a speaker close to the wall may have an effect of + / -3dB at low frequencies. In addition, the disclosed system and method can be used for improved audio upmixing. The various aspects disclosed herein can provide (for example) a circular microphone array with six microphones, which can detect all surrounding walls using the disclosed method. At this time, the disclosed method can determine whether the surrounding wall of the microphone is the left wall or the right wall by comparing the proximity of the wall to each microphone. At this time, the system can perform channel allocation that can be used for upmixing that can be performed automatically. In addition, the disclosed system and method can obtain room characteristics and estimate the distance to the wall or reflector.
[0117] The room impulse response (RIR) generally provides an audio fingerprint of a location in an acoustic environment. RIR can have a variety of applications, such as wall boundary estimation, digital reconstruction of the acoustic environment for professional audio applications, room correction, and frequency response correction of playback systems. The measurement of RIR includes exciting the room (or listening environment), which can be performed by, but not limited to, clapping. The measurement of RIR may also include deconvolution of the audio signal to obtain room characteristics. After exciting the room, RIR may involve reflections. Reverberation may refer to audio reflections reflected back to the audio source. Reverberation is usually not a direct sound, so the reverberation arrives at the microphone later. The reverberation amplitude and return time depend on the material of the surface and the number of reflective areas. The sound continues to reflect until the sound loses its energy due to absorption.
[0118] Fig.24A system 2400 for performing boundary estimation according to one embodiment is depicted. The system 2400 generally includes a first speaker 102a and a second speaker 102b. Each of the speakers 102a, 102b generally includes an audio source 2402. Although only a first speaker 102a and a second speaker 102b are shown, it should be appreciated that any number of speakers may be positioned in a listening environment. The audio source 2402 may be integrated into either of the speakers 102a, 102b to transmit audio from a particular speaker 102 directly into the listening environment 151.
[0119] The first speaker 102a and the second speaker 102b are located away from the wall 2404. In general, it is desirable to know the distance of the first speaker 102a and / or the second speaker 102b from the wall 2404 in the listening environment 151. If one or more of the first speaker 102a and / or the second speaker 102b are placed too close to the wall 2404, this condition may make it difficult for the audio source 2402 to automatically correct the position of the wall 204 relative to the speakers 102a to 102b to ensure that the desired sound field is transmitted into the room (or listening environment 151). In general, the first speaker 102a and / or the second speaker, if positioned too close to the wall 2404, may cause a + / - 3dB effect at low frequencies, respectively. The audio source 2402 (i.e., within the speaker 102a and / or the speaker 102b) can determine the position of the first speaker 102a and / or the second speaker 102b relative to the wall 2404 and employ a correction mechanism to account for the distance of the first speaker 102a and / or the second speaker 102b positioned close to the wall 2404. The system 2400 can improve channel allocation using more than two microphones 106a by employing a correction mechanism to account for the close proximity of the speakers 102a-102b to the wall 2404. The ability to perform channel allocation (e.g., which speaker is left front / right front / rear, etc.) properly enables audio upmixing. It should be appreciated that the audio source 2402 can include any number of controllers 2410 (hereinafter referred to as "controllers 2410") to perform the operations described herein. Although the audio source 2402 can determine the distance of the first speaker 102a and / or the second speaker 102b relative to the wall 2404, it should be recognized that any one or more of the first speaker 102a or the second speaker 102b can also include at least one controller 2412 to determine the distance of the speakers 102a, 102b relative to the wall 2404.
[0120] For example, the controller 2410 may adopt a predetermined measurement scheme such as RIR to provide and transmit an audio fingerprint in the listening environment 151. For example, the controller 2410 may include a driver (not shown) to transmit the audio fingerprint to the listening environment 151. The controller 2410 may also include a memory for storing the audio fingerprint. The system 2400 may adopt various applications of RIR, such as wall boundary estimation, digital reconstruction of the acoustic environment of professional audio applications, room correction, and frequency response correction of the playback system. In one example, the audio source 2402 may excite the room (or listening environment 151) by transmitting an audio signal, and perform, and the measurement of RIR may also include deconvolution of the audio signal to obtain room characteristics. As described above, RIR may involve measuring the captured audio fingerprint (i.e., reflection) after the excitation listening room 151 has been excited. Reverberation may refer to audio reflections reflected back to the audio source 2402. The audio source 2402 may be coupled to the microphones 106a and 106b to receive captured reflections (or reverberations) from the listening environment 151. The reverberations received back by the audio source 2402 are typically not direct sounds, so the reverberations arrive at the microphones 106 later. The amplitude of the reverberations and the time it takes for the reverberations to arrive at the audio source 2402 depends on the number of surface materials and reflective areas within the listening environment 151. The sound continues to reflect until the sound loses its energy due to absorption within the listening environment 151.
[0121] The audio source 2204 may excite the listening environment 151 by transmitting an audio signal comprising an exponential sine sweep (ESS) (or an ESS signal). The ESS signal may be advantageous over the impulse response measurement method because (i) for a signal transmitted with the same length as the length of a maximum length sequence (MLS), the ESS signal has better noise suppression than the MLS method, and (ii) given that the driver transmits the ESS signal directly, the ESS signal may be more robust to nonlinear effects.
[0122] The following equation can be given for the ESS signal:
[0123]
[0124] in:
[0125]
[0126] T represents the duration of the scan. The variable ω 1 and ω 2 Corresponding to the starting and ending frequencies respectively. Due to the frequency variation of the ESS, the energy can be determined at a rate depending on the instantaneous frequency, as given by:
[0127]
[0128] Fig.25 Generally shown is the frequency response of an ESS signal 2500. When the signal 2500 is transmitted from the audio source 2402 to one or more of the first speaker 102a and the second speaker 102b, the ESS signal 2500 includes a peak 2502 thereof.
[0129] The audio source 2402 may measure the RIR by inverse filtering or deconvolution after the first speaker 102a and / or the second speaker 102b plays the EES signal 2500 in the listening environment 151. Then, the controller 2410 applies inverse filtering and extracts the RIR. As described above, the audio source 2402 includes any number of microphones 2420 to record the ESS signal 2500. Then, the audio source 2402 may extract or measure the RIR from the recorded ESS signal 2500. The time-reversed energy of the ESS signal 2500 is reduced, for example, by 3 dB / octave, and the inverse filter has, for example, a 3 dB / octave increase in its energy spectrum to obtain a flat frequency spectrum. Assume that h(t) is the room impulse response, r(t) is the excited room impulse response, and f(t) is the inverse filter.
[0130] h(t)=r(t)*f(t) (13)
[0131] f(t) can be created using post-modulation, which applies an amplitude modulation envelope of +6dB / octave to the spectrum of the time-reversed signal. The general form of the post-modulation function is as follows:
[0132]
[0133] A represents the constant of the modulation function. For time t = 0, ω(t) = w 1 , and for unity gain at time t = 0:
[0134]
[0135] Then, the modulation function becomes:
[0136]
[0137] After modulating the time-reversed signal with m(t), f(t) now has a frequency increase of 3dB / octave. Fig.26 The amplitude spectrum of the inverse filter is shown.
[0138] In general, the measured RIR is obtained by the audio source 2402 by utilizing Equation 13. Thus, aspects associated with Equation 13 correspond to the convolution of the ESS signal and the inverse filter. The audio source 2402 may utilize the measured RIR to estimate the distance of the first speaker 102a and / or the second speaker 102b to the wall 2404. It should be appreciated that the audio source 2402 given the speakers 102a and 102b determines the distance of each speaker 102a and 102b in which the audio source 2402 is positioned. In general, the wall proximity estimate used by the audio source 2402 may be complex because the measured RIR includes reverberation from multiple objects in the listening environment 151 (or room).
[0139] Fig. 27 An example of a curve 2700 corresponding to an RIR measurement performed by an audio source 2402 is generally shown. As shown, the RIR measurement 2700 includes a plurality of peaks 2702. The peaks 2702 may correspond to reflections or reverberations of the ESS signal from various objects in the listening environment 151. A reverberation number 1850 is generally shown at 2704. The reverberation number 1850 generally corresponds to a strong candidate for reverberation of the ESS signal from a wall 2404. This condition may be verified because the amplitude of the peak is highest after a gap 2710 is shown between the highest peak 2712 and the reverberation number 1850 shown at 2704. The highest peak 2172 generally indicates a direct path of the ESS signal from the speaker 102 to the microphone 2420. Additionally, because the peak amplitude associated with 2704 may correspond to the material of the wall 2404, this condition may also be verified. In general, the peak from the wall 2404 may not be obvious, as shown in FIG. Fig. 27 Additionally, the nonlinearity attributed to peak 2702 may be due to a driver (or amplifier) in audio source 2402. For example, an amplifier often produces a false peak in the RIR measurement performed by audio source 2402. Therefore, audio source 2402 may need to account for these conditions when performing RIR measurements.
[0140] The audio source 2402 can overcome the above problems to perform wall distance estimation by (i) sampling or extracting the peaks in the RIR measurement to avoid strong and pseudo peaks (or ringing) close to the peak to be detected around the peak 2702 that may cause erroneous estimates, and / or (ii) scoring each peak to determine the correct peak from the wall 2404. It should be recognized that due to nonlinearity, there are undesirable peaks around the peak 2702, and it is desirable to avoid such peaks in the RIR measurement. In general, the peak 2702 in the RIR measurement may correspond to the direct path from the audio source 2402 to the microphone 2420 and from the reflector to the microphone 2420 on the audio source 2402. Looking closely at the RIR measurement, it can be observed that there is ringing around the peak. The audio source 2402 can extract peaks to detect impulse events. Therefore, in this regard, the audio source 2402 can use a sliding window to extract the peak in each window. The audio source 2402 may find each peak in the window after obtaining the largest peak in the RIR measurement and ignore other peaks in the RIR measurement.
[0141] Fig.28 Generally illustrated is an RIR measurement 2800 with a peak 2802 detected by an audio source 2402, according to one embodiment. Fig.28 It is also shown that, as above combined Fig. 27 The ringing is more pronounced or noticeable. For a sampling rate of 32 kHz, when the distance from the first speaker 102a and / or the second speaker 102b to the wall is 137 cm, the audio source 2402 obtains the RIR measurement 2800. The distance from the speaker to the wall can be obtained based on the following equation:
[0142]
[0143] For example, the “estimated peak index” as set forth above in Equation 17 generally corresponds to the estimated peak value in RIR measurement 2800. Thus, in this regard, Fig.28 As shown, detected peak 2802a corresponds to sample 251, which may be defined as the estimated peak index. The "maximum peak index" typically corresponds to zero. Fig.28 Generally depicted Fig. 27 A modified version of the RIR measurements shown in . For example, Fig. 27 The highest peak 2712 shown in the figure generally corresponds to Fig.28Similarly, the constant value "2" as set forth in Equation 17 generally corresponds to the round trip time. When the audio source 2402 plays the ESS signal, the sound (or ESS signal) contacts the wall 2404 and reflects back to the audio source 2402. Therefore, the ESS signal samples the same distance or path twice. Therefore, in this regard, it is explained why the constant value is set to 2. Fig.28 28. As shown in RIR measurement 2800, if audio source 2402 correctly estimates the peak to be at sample 251, then for a 32kHz sampling rate, audio source 2402 determines the distance between wall 2404 and first speaker 102a or second speaker 102b to be, for example, 137cm using equation 17 above.
[0144] The audio source 2402 tracks the overall trend in the peaks 2802 of the RIR measurements 2800 to estimate the peaks of the reverberation of the RIR measurements 2800. For example, if the ESS signal transmitted by the audio source 2402 does not encounter a wall 2404 or an object in the listening environment 151, the expected trend of the peaks 2802 of the RIR measurements will show or correspond to an overall decrease in the peaks (i.e., a decreasing trend). If the ESS signal transmitted by the audio source 2402 does encounter a wall 2404 or an object in the listening environment 151, the expected trend of the peaks 2802 of the RIR measurements will show a decreasing trend in the peaks 2802, followed by an increasing trend in the peaks, and then followed by a decreasing trend in the peaks 2802. In general, the audio source 2402 stores information corresponding to the peaks 2802 received for the RIR measurements to determine whether there is only a decreasing trend of the peaks 2802 that continuously decreases over time, or whether there is a decreasing trend in the peaks 2802, followed by an increasing peak 2802a. The audio source 2402 may then establish a confidence score calculated by using, for example, the number of negative peaks 2802 multiplied by a percentage increase, such as a value of 1.01. The audio source 2402 may then select a predetermined number of peaks with the highest confidence score (i.e., the largest score) or level (e.g., 20), and then locate the largest peak among the selected peaks 2802. This largest peak may correspond to the peak that exhibits the largest amplitude on the RIR measurement, and may be positive after a long series of decreasing peaks. In this case, the largest peak may be selected as a sample number (e.g., 251), which is then used by the audio source 2402 to insert into equation 17, as provided above, to find the distance of the speaker 102a or 102b from the wall 2404.
[0145] Fig.29A method 2900 for performing boundary estimation involving multiple speakers 102 is depicted according to one implementation. In operation 2902, an audio source 2402 transmits an audio signal in the form of an ESS signal into a listening environment 151. It should be appreciated that the audio source 2402 may transmit the ESS signal from each speaker 102 positioned in the listening environment 151 one at a time, and perform the operations of the method 2900 for each speaker 102 to determine a distance 2404 of the speaker 102 relative to a wall. Each audio source 2402 determines the distance of its corresponding speaker 102a, 102b relative to the wall 2404. It should be appreciated that each speaker 102a, 102b may also transmit the distance information to a mobile device 150 or other device that may require such information so that the mobile device or other audio source may compensate the audio output to mitigate the effect of any one or more of the speakers 102a, 102b being too close to the wall 2404.
[0146] In operation 2904, the audio source 2402 receives reverberation from the listening environment 151 in response to transmitting the ESS signal. In this case, the audio source 2402 detects the peak 2802 of the reverberation in the RIR measurement 2800 and stores information corresponding to the peak 2802 in its memory. In operation 2906, the audio source 2402 performs trend tracking of the peak 2802.
[0147] In operation 2908, the audio source 2402 evaluates the stored peak values 2802 of the reverberation to determine whether there is only a decreasing trend of the peak value 2802 that continuously decreases over time in the RIR measurement, or whether there is a decreasing trend of the peak value 2802 followed by an increasing peak value 2802a in the RIR measurement. If the audio source 2402 determines that the peak value 2802 does not increase over time, the method 2900 moves to operation 2912 and determines that the wall distance of the first speaker 102a or the second speaker 102b cannot be determined. In this case, the method 2900 may move back to operation 2902. If the audio source 2402 determines that there is an increasing peak value 2802a in the RIR measurement, the method 2900 moves to operation 2910.
[0148] In operation 2910, the audio source 2402 establishes a confidence score calculated by using, for example, the number of negative peaks 2802 multiplied by a percentage increase, such as a value of 1.01. The audio source 2402 may then select a predetermined number of peaks with the highest confidence score or level (e.g., 20) and then locate the maximum peak among the selected peaks 2802. Such a maximum peak may correspond to a peak 2802a exhibiting the largest amplitude on the RIR measurement and may be positive after a long series of decreasing peaks 2802. In operation 2912, the audio source 2402 applies the maximum peak to a distance equation (e.g., Equation 17) and also applies other variables as described above in conjunction with Equation 17 to determine the distance of the first speaker 102a or the second speaker 102b relative to the wall 2404.
[0149] It should be recognized that the controller disclosed herein may include various microprocessors, integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variants thereof) and software that cooperate with each other to perform the operations disclosed herein. In addition, such controllers disclosed utilize one or more microprocessors to execute a computer program embodied in a non-transitory computer-readable medium, the computer program being programmed to perform any number of functions disclosed. In addition, the controller as provided herein includes a housing and various numbers of microprocessors, integrated circuits, and memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) positioned within the housing. The disclosed controller also includes hardware-based inputs and outputs for receiving data from and transmitting data to other hardware-based devices discussed herein, respectively.
[0150] Although the above describes exemplary embodiments, it is not meant that these embodiments describe all possible forms of the present invention. On the contrary, the words used in the specification are words for description rather than limitation, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. In addition, the features of the various embodiments realized can be combined to form other embodiments of the present invention.
Claims
1. An audio system, wherein include: a plurality of speakers positionable in a listening environment and arranged to transmit audio signals in the listening environment, each speaker being programmed to determine angle information of the audio signal received at the speaker relative to other speakers of the plurality of speakers and to transmit a first signal indicative of the angle information; as well as A mobile device is programmed to receive the first signal from each of the speakers and determine a position of each speaker in the listening environment based at least on the angular information.
2. The audio system of claim 1, wherein each of the plurality of speakers comprises at least two microphones located on each speaker and spaced apart from each other to receive the audio signals from the other speakers of the plurality of speakers.
3. The audio system of claim 2, wherein the mobile device is further programmed to store the angle information and determine the orientation of the at least two microphones of each speaker relative to each other based at least on the angle information.
4. The audio system of claim 3 , wherein the mobile device is further programmed to determine, based at least on the angles of the plurality of speakers, that the orientation of the at least two microphones for each speaker is one of: located in the same plane relative to the other speakers, or located in a different plane relative to the other speakers for at least one of the speakers.
5. An audio system according to claim 3, wherein the mobile device is also programmed to determine that any one or more of the multiple speakers corresponds to an outlier based at least on the angle information exceeding a predetermined threshold, the outlier indicating the speaker providing erroneous angle information or an obstacle existing near the speaker.
6. The audio system of claim 5, wherein the mobile device is further programmed to prompt a user to move closer to the obstacle located near the speaker determined to be the outlier.
7. The audio system of claim 3, wherein the mobile device is further programmed to select one or more speakers of the plurality of speakers as reference speakers and arbitrarily select reference coordinates of the other speakers relative to the reference speakers.
8. The audio system of claim 7, wherein the mobile device is further programmed to determine an estimated distance coordinate for each of the other speakers based at least on the angle information and the reference coordinates.
9. The audio system of claim 8, wherein the mobile device is further programmed to expand the candidate coordinate estimate of at least one speaker of the plurality of speakers and provide an expanded angle to provide an intersection point between at least two of the speakers.
10. The audio system of claim 9, wherein the mobile device is further programmed to determine an error based at least on the expanded candidate coordinate estimates and an angle based on the expanded coordinate estimates to select a final candidate for the position of the at least one speaker.
11. A method, wherein include: transmitting an audio signal in a listening environment via a plurality of loudspeakers positionable in the listening environment, determining, by each speaker, angle information of the audio signal received at each speaker relative to other speakers of the plurality of speakers, and transmitting a first signal indicating the angle information; as well as The first signal is received at a mobile device from each of the speakers and a position of each speaker in the listening environment is determined based at least on the angle information.
12. The method of claim 11, wherein each speaker of the plurality of speakers comprises at least two microphones located on each speaker and spaced apart from each other to receive the audio signals from the other speakers of the plurality of speakers.
13. The method of claim 12, further comprising storing the angle information at the mobile device; and An orientation of the at least two microphones of each loudspeaker relative to each other is determined based at least on the angular information.
14. The method of claim 13, further comprising determining, based at least on the angles of the plurality of speakers, that the orientation of the at least two microphones of each speaker is one of: located in the same plane relative to the other speakers, or located in a different plane relative to the other speakers for at least one of the speakers.
15. The method according to claim 13 also includes determining that any one or more of the multiple speakers corresponds to an outlier based at least on the angle information exceeding a predetermined threshold, the outlier indicating the speaker providing erroneous angle information or an obstacle existing near the speaker. 16 . The method of claim 15 , further comprising prompting a user via the mobile device to move closer to the obstacle located near the speaker determined to be the outlier.
17. The method of claim 13, further comprising selecting one of the plurality of speakers as a reference speaker, and arbitrarily selecting reference coordinates of the other speakers relative to the reference speaker.
18. The method of claim 17, further comprising determining an estimated distance coordinate for each of the other speakers based at least on the angle information and the reference coordinates.
19. The method of claim 18, further comprising extending the candidate coordinate estimate for at least one of the plurality of speakers and providing an extended angle to provide an intersection point between at least two of the speakers.
20. The method of claim 19, further comprising determining an error based at least on the expanded candidate coordinate estimates and an angle based on the expanded coordinate estimates to select a final candidate for the position of the at least one speaker.
21. An audio system, wherein include: a plurality of speakers positionable in a listening environment and arranged to transmit audio signals in the listening environment, each speaker being programmed to determine angle information of the audio signal received at the speaker relative to other speakers of the plurality of speakers and to transmit a first signal indicative of the angle information; as well as A master speaker is programmed to receive the first signal from each of the speakers and determine a position of each speaker in the listening environment based at least on the angular information.
