Sample-accurate delay identification in the frequency domain

Abstract

Systems, methods, and computer program products for sample-accurate delay identification in the frequency domain. In some embodiments, estimation is performed on a first data block indicating samples of a first audio signal and a second data block indicating samples of a second audio signal, and includes determining a coarse delay estimate, including by determining a gain of an estimate of one of the first blocks when applied to some of the second blocks, and identifying one of the estimates that has the best spectral match with the one of the first blocks. An optimized delay estimate is determined based on the coarse estimate and some of the gains. Optionally, at least one metric is generated that indicates a confidence level in the optimized delay estimate. Audio processing (e.g., echo cancellation) may be performed on the frequency domain data, including by performing time alignment based on the optimized delay estimate.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 901,345, filed on September 17, 2019, entitled Sample Accurate Delay Identification in the Frequency Domain. Technical Field

[0003] The present disclosure relates generally to audio signal processing. Some embodiments relate to estimating a time delay applied to an audio signal relative to another audio signal in order to time-align the signals (eg, to perform echo cancellation or another audio processing on the signals). Background Art

[0004] Echo cancellation techniques produce problematic outputs when the microphone signal precedes the echo signal, and echo cancellation techniques generally work better when the microphone input signal and the echo signal are approximately time aligned. It would be useful to implement a system that can identify time delays between signals (i.e., a time delay that should be applied to one signal relative to another signal in order to time align the signals) to improve the implementation of echo cancellation (or another audio processing) on ​​the signals.

[0005] The echo cancellation system can operate in the time domain on a time domain input signal. If there are many audio samples (e.g., tens of thousands of audio samples), implementing such a system can be extremely complex, especially if long time domain correlation filters are used, and may not produce good results.

[0006] Alternatively, the echo cancellation system may operate in the frequency domain on a frequency transformed representation of each time domain input signal (i.e., rather than operating in the time domain). Such a system may operate on a set of complex-valued bandpass representations of each input signal (which may be obtained by applying an STFT or another complex-valued uniformly modulated filter bank to each input signal). For example, U.S. Patent Application Publication No. 2019 / 0156852, published on May 23, 2019, describes echo management (echo cancellation or echo suppression) that includes an estimated (in the frequency domain) delay between two input audio streams. Echo management (including delay estimation) implements adaptation of a set of prediction filters.

[0007] However, the need to adapt a set of prediction filters (eg, using a gradient descent adaptive filter method) makes estimating the time delay between audio signals more complicated. It would be more useful to estimate the time delay between audio signals in the frequency domain without performing adaptation of the prediction filters.

[0008] Symbols and nomenclature

[0009] Throughout this disclosure, including in the claims, the term "heuristic" is used to mean based on trial and error (e.g., obtains good results at least under expected or typical conditions) or determined experimentally (e.g., obtains good results at least under expected or typical conditions). For example, a "heuristic" value (e.g., a parameter or metric) can be determined experimentally (e.g., by adjustment), or can be determined by a simplified method that typically only determines an approximate value, but determines the value with sufficient accuracy in the relevant use case. For another example, a "heuristic" value for processing data can be determined by at least one statistical feature of the data that is expected (based on trial and error or experimentation) to obtain good results in the expected use case. For another example, a metric (e.g., a confidence metric) can be called a "heuristic" metric if the metric has been determined based on trial and error or experimentation to obtain good results at least under expected or typical conditions.

[0010] Throughout this disclosure, including in the claims, the term "delay" of (or between) two audio signals (e.g., time-domain audio signals, or frequency-domain audio signals generated by transforming time-domain audio signals) is used to refer to a time delay that should be applied to one signal relative to another signal in order to time-align the signals.

[0011] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, adjusting, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has been initially filtered or preprocessed before the operation is performed).

[0012] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to an apparatus, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and another XM inputs are received from external sources) may also be referred to as a decoder system.

[0013] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio data). Examples of processors include a field programmable gate array (or another configurable integrated circuit or chipset), a digital signal processor programmed and / or otherwise configured to perform pipelined processing on audio data, a graphics processing unit (GPU) configured to perform processing on audio data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chipset.

[0014] Throughout this disclosure, including in the claims, the term "couple" is used to mean either a direct or indirect connection. Thus, if a first device is referred to as being coupled to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0015] Throughout this disclosure, including in the claims, "audio data" means data indicative of sounds (e.g., speech) captured by at least one microphone, or data generated (e.g., synthesized) so that the data can be rendered as sounds (e.g., speech) played (through at least one speaker). For example, audio data can be generated to be used as a substitute for data indicative of sounds (e.g., speech) captured by at least one microphone. Summary of the invention

[0016] One class of embodiments of the present invention is a method for estimating a time delay between audio signals using a frequency transformed representation of each of the signals (e.g., in a frequency domain audio signal generated by transforming a time domain input audio signal). The estimated time delay is an estimate of a time delay that should be applied to one audio signal (e.g., a pre-transformed time domain audio signal) relative to another audio signal (including any time delay applied to the other signal) in order to time align the signals, e.g., in order to perform a desired audio processing (e.g., echo cancellation) on at least one of the two signals. In a typical embodiment, the time delay estimation is performed on a complex-valued frequency bandpass representation of each input signal (which may be obtained by applying an STFT or another complex-valued uniformly modulated filter bank to each input signal). Typical embodiments that perform the time delay estimation do not require performing adaptation of the prediction filter.

[0017] Some embodiments of a delay estimation method are performed on a first sequence of frequency domain data blocks M(t, k) of audio samples of a first audio signal (e.g., a microphone signal) and a second sequence of frequency domain data blocks P(t, k) of audio samples of a second audio signal (e.g., a playback signal) to estimate the delay between the first audio signal and the second audio signal, wherein t is an index representing time and k is an index representing a frequency band, the method comprising the following steps:

[0018] (a) for each block P(t,k) in the second sequence, where t is an index representing the time of each block and k is an index representing the frequency band, providing a delay block P(t,b,k), where b is an index representing the block delay time, wherein each value of the index b is an integer number of block delay times of a corresponding one of the delay blocks delayed relative to the time t;

[0019] (b) For each block M(t,k), determine a rough estimate of the delay at time t, b best(t), comprising determining an estimate M of block M(t,k) when applied to each of the delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t,b,k) is the one with the best spectral match with the block M(t,k), where the rough estimate value b best (t) has an accuracy of approximately one in the block delay times; and

[0020] (c) Based on a rough estimate of b best (t) and some of the gains (e.g., using characteristics of the time-domain to frequency-domain transform that has been used to generate blocks M(t, k) and P(t, k)) to determine an optimized estimate of the delay R(t) at time t (e.g., R(t) = L med (t), as in the example embodiments described herein), wherein the optimized estimate R(t) has an accuracy of approximately audio sample times (e.g., in the case where the frequency domain data has been generated by applying a time-domain to frequency domain transform to the time domain data, the audio sample times are the sample times of the pre-transformed data).

[0021] In some embodiments, at least one of a rough estimate or an optimized estimate of the delay is determined using one or more heuristically determined parameters. For example, in some embodiments, step (b) comprises determining, for each of the delay blocks P(t,b,k), a heuristic unreliability factor U(t,b,k) on a per-band basis (e.g., for a selected subset of the full set of windows k). In some such embodiments, the gain H(t,b,k) is a gain for each of the delay blocks P(t,b,k), and each of the unreliability factors U(t,b,k) is determined according to a set of statistics, the set comprising: an average value H m (t, b, k), which is determined by averaging two times (time t and previous time t-1) according to the gain H(t, b, k); and the variance value H v (t,b,k), which is calculated by averaging the time t and t-1 according to the gain H(t,b,k) and the average value H m (t,b,k) is determined.

[0022] In some embodiments, step (b) comprises determining a goodness factor Q(t,b), which may be calculated for each value of the estimate M at time t and index b. est (t,b,k) is determined heuristically, and a rough estimate of b is determined best (t) includes selecting the best (eg, smallest) of the goodness factors Q(t,b).

[0023] In some embodiments, the method further comprises the steps of: (d) applying a thresholding test to determine whether a candidate optimized estimate of the time delay (e.g., the most recently determined value L(t), as in some example embodiments described herein) should be used to update a previously determined optimized estimate of the time delay R(t); and (e) updating the previously determined optimized estimate of the time delay R(t) using the candidate optimized estimate only if the thresholding test determines that the thresholding condition is satisfied. Typically, step (d) comprises determining a value for the coarse estimate b best (t), a set of smoothing gains H s (t,b best (t), k) should be considered as a gain candidate set for determining an updated optimized estimate of the time delay. In some embodiments including steps (d) and (e), the method further includes determining a fourth best rough estimate of the time delay at time t. 4thbest (t) and

[0024] Step (b) involves estimating the value M for each value of time t and index b est (t,b,k), determine the goodness factor Q(t,b), and determine the rough estimate b best (t) includes selecting the best (e.g., the smallest) of the goodness factors Q(t,b), and

[0025] Step (d) comprises providing a rough estimate of b best The goodness factor Q(t,b best ), for the fourth best rough estimate b 4thbest The goodness factor Q(t,b 4thbest ) and for a rough estimate of b best The estimated value of (t) is M est (t,b best ,k) Apply threshold processing test.

[0026] For example, multiple sets of gain H that meet the threshold processing conditions can be used for the sequence of time t. s (t,b best (t), k) determine an optimized estimate R(t), and step (e) may include identifying a median value of a set of X (e.g., X=40) values ​​as the optimized estimate R(t) of the delay, wherein the X values ​​include the most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the delay.

[0027] Typical embodiments of the present invention do not require the use of a separate time-domain correlation filter, but instead attempt to estimate the delay in the frequency domain where the intended audio processing is being (or will be) performed. Typically, the expected estimated delay (between two audio signals) is used to time-align the signals so that the intended audio processing (e.g., echo cancellation) can be performed on the aligned signals. For example, the intended audio processing can be performed on the output of a DFT modulated filter bank (e.g., STFT or another uniformly modulated complex filter bank), which is a common signal representation used in audio processing systems, so performing the delay estimation in the same domain as the intended audio processing reduces the complexity required for the delay estimation.

[0028] Some embodiments estimate the delay with an accuracy of approximately the individual sample times of a pre-transformed (time domain) version of the input signal. For example, some embodiments implement a first stage of determining the delay coarsely (accurately to a block of frequency domain data that has been generated by applying a time domain to frequency domain transform to the input signal) and a second stage of determining a sample accurate delay based in part on the coarse delay determined in the first stage.

[0029] Some embodiments also generate at least one confidence metric that indicates confidence in the accuracy of the estimated delay. For example, the confidence metric can be generated using statistical data over a period of time to provide at least one indication of whether the delay calculated at the current time is credible. For example, the confidence metric can be used to indicate whether the estimated delay is incorrect to the extent that it cannot be corrected, such that other operations (e.g., deactivating an acoustic echo canceller) or audio processing functions should be performed.

[0030] Aspects of the present invention include a system configured (e.g., programmed) to perform any embodiment of the method of the present invention or its steps, and a tangible, non-transitory computer-readable medium (e.g., a disk or another tangible storage medium) implementing non-transitory storage of data, the medium storing code for performing any embodiment of the method of the present invention or its steps (e.g., code that can be executed to perform any embodiment of the method of the present invention or its steps). For example, an embodiment of the system of the present invention may be or include a programmable general-purpose processor, a digital signal processor, a GPU, or a microprocessor that is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations on data, including embodiments of the method of the present invention or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform an embodiment of the method of the present invention (or its steps) in response to asserted data. Some embodiments of the system of the present invention may be implemented (or have been implemented) as a cloud service (e.g., where the elements of the system are in different locations and data is transferred between these locations, for example, via the Internet). BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram of an embodiment of the time delay estimation system of the present invention integrated into a communication system.

[0032] Figure 2 is a block diagram of an example system configured to perform delay identification in the frequency domain.

[0033] Figure 3 is a graph showing the performance resulting from data reduction selecting regions of contiguous frequency bins k relative to data reduction selecting prime frequency bin values ​​k (according to some embodiments of the present invention).

[0034] Figure 4 is a flow chart of an example process for delay identification in the frequency domain.

[0035] Figure 5 It is used for implementation reference Figures 1 to 4 Describes the features and processes of a mobile device architecture. DETAILED DESCRIPTION

[0036] Figure 1 is a block diagram of an embodiment of the time delay estimation system of the present invention integrated into a communication system. Figure 1 The communication system 2 of can be a communication device including a processing subsystem (at least one processor programmed or otherwise configured to implement a communication application 3 and an audio processing object 4) and physical device hardware 5 coupled to the processing subsystem (including a loudspeaker 16 and a microphone 17). Typically, the system 2 includes a non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform an embodiment of the method of the present invention.

[0037] An audio processing object (APO) 4 is implemented (i.e., at least one processor is programmed to execute APO 4) to perform an embodiment of the method of the present invention to estimate a delay between two audio streams, where a delay is a time delay that should be applied to one stream relative to another stream in order to time align the streams. As implemented in system 2, the audio streams are: a playback audio stream (audio signal) provided to a loudspeaker 16, and a microphone audio stream (audio signal) output from a microphone 17. APO 4 is also implemented (i.e., it includes an implemented speech processing subsystem 15) to perform audio processing (e.g., echo cancellation and / or another audio processing) on ​​the audio streams. Although subsystem 15 is identified as a speech processing subsystem, it is contemplated that in some embodiments, subsystem 15 performs audio processing other than speech processing (e.g., pre-processing for communication application 3 or another audio application, which may or may not include echo cancellation). Detection of the delay between the streams is performed in accordance with an exemplary embodiment of the present invention (e.g., in an environment where the delay cannot be known in advance) in an effort to ensure that the audio processing (e.g., echo cancellation) performed by subsystem 15 will operate appropriately.

[0038] APO 4 may be implemented as a software plug-in and interact with audio data present in the processing subsystem of system 2. The delay estimation performed by APO 4 may provide a robust mechanism for identifying the delay between the microphone audio stream (the "capture stream" processed by APO 4) and the "loopback" stream (which contains the audio data output from the communication application 3 for playback by the loudspeaker 16), thereby ensuring that the echo cancellation (or other audio processing) performed by subsystem 15 (as well as the audio processing performed by application 3) will operate properly.

[0039] exist Figure 1 4, APO 4 processes M channels of audio samples of the microphone output stream block by block, and processes N channels of audio samples of the playback audio stream block by block. In a typical embodiment, the delay estimation subsystem 14 of APO 4 estimates the delay between the streams with per-sample accuracy (i.e., the delay estimate is accurate to approximately individual pre-transformed audio sample times (i.e., the sample times of the audio before transformation in subsystems 12 and 13), rather than just to approximately individual blocks of samples).

[0040] In a typical implementation, APO 4 (i.e., delay estimation subsystem 14 of APO 4) estimates delay in the signal domain in which audio processing (e.g., in subsystem 15) has been performed. For example, subsystems 14 and 15 both operate on frequency domain data output from time domain to frequency domain transform subsystems 12 and 13. Subsystems 12 and 13 may both be implemented as DFT modulated filter banks (e.g., STFT or another uniformly modulated complex filter bank) so that the signals output therefrom have a signal representation commonly employed in audio processing systems (e.g., a typical implementation of subsystem 15), and having delay estimation performed in this domain reduces the complexity required to implement APO 4 to perform delay estimation (in subsystem 14) as well as audio processing in subsystem 15.

[0041] The exemplary embodiments described herein (eg, by Figure 1 Delay estimation, as performed in a typical implementation of APO 4, is a method for robustly (and typically both efficiently and reliably) identifying the delay of an input audio signal or the delay between input audio signals using a frequency domain representation of the input audio signal with an accuracy of approximately the audio sample time of the frequency domain audio data. Such embodiments typically operate in a block audio domain (e.g., a complex-valued block transform domain) in which there is a frequency domain audio data stream comprising blocks of frequency domain audio data. The estimated delay is an estimate of the time delay that should be applied to one signal relative to another signal in order to time-align the signals, and can be used to compensate for the time delay between two audio sources. Some embodiments also generate at least one "confidence" metric (e.g., one or more of the heuristic confidence metrics C1(t), C2(t), and C(t) described below) that indicates the confidence that the delay estimate is accurate at a given point in time. Confidence metrics (sometimes called confidence metrics) can be used to correct for latency variations in the system (if the latency is dynamic) or to notify the system that an operating state or condition is not ideal and should perhaps be adapted to some extent (e.g., by disabling features being implemented by the system).

[0042] like Figure 1 As indicated in FIG. 4 , APO 4 includes (implements) delay lines 10 and 11, time-domain to frequency-domain transform subsystems 12 and 13, a delay estimation subsystem 14, and a speech processing subsystem 15. Delay line 10 stores the last N1 blocks of time-domain playback audio data from application 3, and delay line 11 stores the last N2 blocks of time-domain microphone data, where N1 and N2 are integers, and N1 is greater than N2.

[0043] The time domain to frequency domain transform subsystem 12 transforms each playback audio data block output from line 10 and provides the resulting frequency domain playback audio data block to a delay estimation subsystem 14. In a typical embodiment, APO 4 (e.g., its subsystem 12) implements data reduction in which only a subset of the full set of frequency bands (subbands) of the frequency domain playback audio data is selected and only the audio in the selected subband subset is used to perform delay (time delay) estimation.

[0044] The time domain to frequency domain transform subsystem 13 transforms each microphone data block output from line 11 and provides the resulting frequency domain microphone data block to the delay estimation subsystem 14. In a typical embodiment, APO 4 (e.g., its subsystem 13) implements data reduction in which only a subset of the full set of frequency bands (subbands) of the frequency domain playback audio data is selected, and only the audio in the selected subband subset is used to perform delay (time delay) estimation.

[0045] Subsystem 14 of APO 4 estimates the time delay between the microphone and the playback audio stream. Some embodiments of the delay estimation method are performed on a first sequence of blocks M(t, k) of frequency domain microphone signals (output from transformation subsystem 13) and a second sequence of blocks P(t, k) of frequency domain playback audio data (output from transformation subsystem 12), where t is an index representing the time of each block and k is an index representing the frequency band. In these embodiments, the method comprises:

[0046] (a) for each block P(t, k) in the second sequence, providing a delay block P(t, b, k), where b is an index representing a block delay time, where each value of the index b is an integer number of block delay times of a corresponding one of the delay blocks that is delayed relative to time t (e.g., for each playback audio data block input to delay line 10, transform subsystem 12 provides N1-N2 delay blocks P(t, b, k) to subsystem 14, each delay block having a different value of index b. Each playback audio data block input to delay line 10 corresponds to a microphone data block M(t, k) input to delay line 11); and

[0047] (b) For each block M(t,k), determine a rough estimate of the delay at time t, b best (t), comprising determining an estimate M of block M(t,k) when applied to each of the delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t,b,k) is the one with the best spectral match with the block M(t,k), where the rough estimate value b best (t) has an accuracy of approximately one in the block delay times; and

[0048] (c) Based on a rough estimate of bbest (t) and some of the gains (e.g., using the characteristics of the time-domain to frequency-domain transforms that have been applied in subsystems 12 and 13 to generate blocks M(t, k) and blocks P(t, k)) determine an optimized estimate of the delay R(t) at time t (e.g., R(t) = L med (t), as hereinafter referred to Figure 2 In the example embodiment described herein, the optimized estimate R(t) has an accuracy of approximately one audio sample time.

[0049] In some embodiments, subsystem 14 uses a heuristic method to determine the rough estimate b best (t). For example, in some embodiments, performing step (b) by subsystem 14 includes determining, for each of the delay blocks P(t,b,k), a heuristic unreliability factor U(t,b,k) on a per-band basis (e.g., for a selected subset of the full set of windows k). In some such embodiments, the gain H(t,b,k) is a gain for each of the delay blocks P(t,b,k), and each of the unreliability factors U(t,b,k) is determined according to a set of statistical values, the set comprising: an average value H m (t, b, k), which is determined by averaging two times (time t and previous time t-1) according to the gain H(t, b, k); and the variance value H v (t, b, k), which is obtained by averaging the two times according to the gain H(t, b, k) and the average value H m (t,b,k) is determined.

[0050] In some embodiments, step (b) performed by subsystem 14 includes determining an estimate M for each value of time t and index b. est The goodness factor Q(t,b) of (t,b,k) and the rough estimate b are determined best (t) includes selecting the best (eg, smallest) of the goodness factors Q(t,b), for example, as described below with reference to Figure 2 described.

[0051] During execution of some embodiments of the method, subsystem 14 also performs the following steps:

[0052] (d) Applying a thresholding test to determine candidate optimized estimates of the delay (e.g., as described below with reference to Figure 2 whether the most recently determined value L(t) described should be used to update the previously determined optimized estimate R(t) of the delay; and

[0053] (e) Updating the previously determined optimized estimate of the time delay R(t) using the candidate optimized estimate only if the thresholding test determines that the thresholding condition is satisfied.

[0054] References below Figure 2 Example implementations of steps (d) and (e) are described. Typically, step (d) comprises determining (in subsystem 14) a value for the rough estimate b best (t), a set of smoothing gains H s (t,b best (t), whether k) should be considered as a gain candidate set for determining the updated optimized estimate of the delay.

[0055] In some embodiments including steps (d) and (e), the method further comprises determining a fourth best rough estimate b of the delay at time t 4thbest (t) and

[0056] Step (b) involves estimating the value M for each value of time t and index b est (t,b,k), determine the goodness factor Q(t,b), and determine the rough estimate b best (t) includes selecting the best (e.g., the smallest) of the goodness factors Q(t,b), and

[0057] Step (d) comprises providing a rough estimate of b best The goodness factor Q(t,b best ), for the fourth best rough estimate b 4thbest The goodness factor Q(t,b 4thbest ) and for a rough estimate of b best The estimated value of (t) is M est (t,b best ,k) Apply threshold processing test.

[0058] For example, multiple sets of gain H that meet the threshold processing conditions can be used for the sequence of time t. s (t,b best (t), k) determine an optimized estimate R(t), and step (e) may include identifying a median value of a set of X (e.g., X=40) values ​​as the optimized estimate R(t) of the delay, wherein the X values ​​include the most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the delay.

[0059] During execution of some embodiments of the method, subsystem 14 also generates and outputs (e.g., provides to subsystem 15) at least one confidence metric indicating confidence in the accuracy of the estimated delay. For example, the confidence metric can be generated using statistical data over a period of time to provide at least one indication of whether the delay calculated at the current time is credible. For example, the confidence metric can be used to indicate whether the estimated delay is not credible, such that other operations (e.g., deactivating an acoustic echo canceller) or audio processing functions should be performed. hereinafter referred to as Figure 2 Describes an example of confidence metric generation.

[0060] Figure 2 is a block diagram of an example system 200 configured to perform delay identification in the frequency domain. Figure 2 The system is coupled to (eg, includes) a microphone 90, a loudspeaker 91, and two time-domain to frequency-domain transform subsystems 108 and 108A, which are coupled as shown. Figure 2 The system comprises a delay estimator 93, a preprocessing subsystem 109 and a frequency domain to time domain transformation subsystem 110, which are coupled as shown. Figure 2 (not shown) may apply an adjustable time delay to each audio stream to be input to the time domain to frequency domain transform subsystem 108, for example, when Figure 2 The elements shown in are included in a system configured to implement delay adjustment.

[0061] Considered together, the preprocessing subsystem 109 and the frequency domain to time domain transformation subsystem 110 are Figure 1 The time domain audio signal output from the subsystem 110 is an example embodiment of the speech processing system 15 of FIG. 1 . The time domain audio signal output from the subsystem 110 is a signal that can be provided to a communication application (e.g., Figure 1 Optionally, a processed version of the playback audio signal is also output from subsystem 110.

[0062] The delay estimator 93 (by Figure 2 The data reduction subsystems 103, 103A, 101, 102, 111, 105, 106 and 107 (indicated by the dashed box in FIG. 1 ) contain subsystems 103, 103A, 101, 102, 111, 105, 106 and 107 which will be described below. The input to the data reduction subsystems 103 and 103A is a complex-valued transform domain (frequency domain) representation of the two audio data streams. Figure 2In the example shown (but not in other contemplated embodiments of delay estimation according to the present invention), a time-domain playback audio stream is provided as an input to the loudspeaker 91 and to the input of the transform subsystem 108A, and the output of the subsystem 108A is one of the frequency-domain audio data streams provided to the delay estimator 93. In the example, the other frequency-domain audio data stream provided to the delay estimator 93 is the audio stream output from the microphone 90, which has been transformed into the frequency domain by the transform subsystem 108. In the example, the microphone audio data (the output of the microphone 90 that has been transformed in the time / frequency domain in the subsystem 108) is sometimes referred to as the first audio stream, and the playback audio data is sometimes referred to as the second audio stream.

[0063] The delay estimator (delay estimation subsystem) 93 is configured to calculate (and provide to the pre-processing subsystem 109) a delay estimate (i.e., data indicating the time delay between two audio data streams input to the subsystem 93 with an accuracy of approximately individual sample times), and at least one confidence measure regarding the delay estimate. Figure 2 In the embodiment of the present invention (and other typical embodiments of the present invention), the delay estimation is performed in two stages. The first stage roughly determines the delay with an accuracy of about the frequency domain data block input to subsystem 93 (i.e., a rough delay estimate b of the output time t of subsystem 111 of subsystem 93). best The second stage determines the sample-accurate delay (i.e., the optimized delay estimate L of the output time t of subsystem 107 of subsystem 93). med (t)), the sample-precision delay is based in part on the coarse delay determined in the first stage.

[0064] The time domain to frequency domain transform subsystem 108 transforms each microphone data block and provides the resulting frequency domain microphone data block to the data reduction subsystem 103. Subsystem 103 performs data reduction, wherein only a subset of frequency bands (subbands) of the frequency domain microphone audio data is selected and only the selected subset of subbands is used for delay estimation. The following aspects of a typical implementation of data reduction will be described.

[0065] The time domain to frequency domain conversion subsystem 108A converts each playback audio data block and provides the resulting frequency domain playback audio data block to the data reduction subsystem 103A. Subsystem 103A performs data reduction in which only a subset of frequency bands (subbands) of the frequency domain playback audio data is selected and only the selected subset of subbands is used for delay estimation. The following aspects of a typical implementation of data reduction will be described.

[0066] Subsystem 111 (in Figure 2 The "Compute Gain Map and Statistics" subsystem in the best(t)), and outputs a rough delay estimate to the subsystem 106. best (t) of the delay block (in delay line 102), subsystem 111 also generates and outputs a gain value H to subsystem 105. s (t,b best (t),k).

[0067] The inverse transform and peak determination subsystem 105 generates the gain value H(t,b best , k) performs an inverse transform (described in detail below) and determines the peak value of the value produced by this inverse transform. This peak value, i.e., the value discussed below, is provided to subsystem 106.

[0068] The combination subsystem 106 calculates the rough estimate b best The estimated value L(t) is provided to subsystem 107 as described below.

[0069] Subsystem 107 (in Figure 2 The final (optimized) delay estimate L is determined based on the estimated value L(t). med (t), as described below. Under some conditions (described below), the median of the X (e.g., X=40) most recent values ​​of L(t) is the final (optimized) delay estimate L med (t). Subsystem 107 also generates one or more heuristic confidence metrics (eg, confidence metrics C1(t), C2(t), and C(t) described below). The final delay estimate and each confidence metric are provided to pre-processing subsystem 109.

[0070] Next, we describe in more detail Figure 2 components of the system.

[0071] ( Figure 2The data reduction subsystems 103 and 103A of the present invention filter the frequency domain audio stream entering the delay estimation subsystem 93. Specifically, each of the subsystems 103 and 103A selects a subset of the frequency bands (subbands) of the audio data input thereto. Subsystem 103 provides each block of the selected subband of the microphone signal to the delay line 101. Subsystem 103A provides each block of the selected subband of the playback signal to the delay line 102. The selected subbands are generally at frequencies that are well captured and reproduced by a known system (e.g., the system's microphone 90 and loudspeaker 91). For example, if the system is implemented in or on a device with a smaller speaker, the selected subset may exclude frequencies corresponding to low-frequency information. The indices of the selected subbands are not necessarily consecutive, but rather it is generally advantageous for the subbands to have a certain diversity (as will be described below). The number of subbands selected (and thus the number of corresponding frequency band indices used for delay estimation) may be equal to or substantially equal to 5% of the total number of frequency subbands of the data streams output from each of subsystems 108 and 108A.

[0072] Figure 2 Subsystem 93 of stores the last N1 blocks of the data-reduced first audio stream (data-reduced microphone data) in delay line 101, where N1 is a tuning parameter. In the example, N1=20. The number N1 can be based on the configuration of each filter bank employed in the relevant implementation of subsystem 108, where the number of blocks (e.g., N1=20) is selected so that delay line 101 stores a desired amount of audio data (e.g., at least about 400 milliseconds of audio data). Other values ​​of N1 are possible. Introducing a delay using delay line 101 allows the system to detect acausality, which may occur when a given signal appears in the microphone data before appearing in the playback data. Acausality can occur in a system where (for example) additional processing blocks are employed (in Figure 2 ) processes the playback audio provided to the loudspeaker (e.g., before it is transformed in the associated time-domain to frequency-domain transform subsystem 108), and the delay estimation subsystem 93 is unaware of (e.g., has no way of knowing) such additional processing.

[0073] Subsystem 93 also implements a delay line 102 for storing the last N2 blocks of the data-reduced second audio stream (data-reduced playback data). The length of delay line 102 is equal to N2 blocks, where N2 is (at least approximately) equal to twice the length of microphone delay line 101 (N1 blocks). In the example where N1=20 blocks, N2=40 blocks is an example of a tuning parameter N2. Other values ​​of N2 are possible.

[0074] For each block of delayed audio in line 102, Figure 2Subsystem 111 of the system calculates a set of gains for the longest delay block M(t,k) that maps the playback audio P(b,k) to the microphone data in line 101:

[0075]

[0076] Where t represents the time point at which the delay estimation subsystem 93 is called, and is incremented with each call to the delay estimation system; b represents the block index of each data block in the delay line 102; and k represents the frequency band. The real-valued parameter ε is used for two purposes: to prevent division by zero when the playback audio is zero, and to set a threshold above which we do not wish to compute reliable gains.

[0077] In situations where one audio stream is only partially correlated with the other audio stream (e.g., in the case of duplex communication, during double talk or near-end only talk), the calculated gain (H(t,b,k)) may be invalid. To help identify whether the gain is valid, the subsystem 111 preferably calculates some statistics on a per-band basis. Specifically, the subsystem 111 calculates the mean and variance estimates for each gain for each block:

[0078] H m (t,b,k)=αH m (t-1,b,k)+(1-α)H(t,b,k)

[0079] H vinst (t,b,k)=|H(t,b,k)-H m (t-1,b,k)| 2

[0080] H v (t,b,k)=βH v (t-1,b,k)+(1-β)H vinst (t,b,k)

[0081] If the variance is very small, then we can conclude that the microphone audio M and the playback audio P are closely related, and P is much larger than ε. If the variance is large, then we can conclude that P is much smaller than ε and the variance is the variance of M / ε, or that there is not a good correlation between P and M.

[0082] Subsystem 111 encodes these values ​​into a heuristic "unreliability factor" for each gain:

[0083]

[0084] This expression can be shown to vary between 0 (indicating an excellent mapping between M and P) and 1 (indicating a poor mapping between M and P). A thresholding operation (where ρ is a threshold) is performed on U(t,b,k) to determine whether each gain H(t,b,k) should be smoothed into a set of actual mapping estimates, and smoothing is only performed on valid and reliable gains. The following equation describes a thresholding operation (where ρ is a threshold) on U(t,b,k) to determine whether a gain H(t,b,k) should be used to generate a set of smoothed gains H s (t,b,k), which are used to determine the microphone signal estimate M est (t,b,k), where the smoothing is done over time for all time intervals where U(t,b,k) is below the threshold:

[0085]

[0086] As part of the tuning process, ρ is chosen. An example value is ρ = 0.05.

[0087] Once this process is complete, subsystem 111 determines an estimate of the microphone signal based on the smoothed gain of each delay gain block:

[0088] M est (t,b,k)=H s (t,b,k)P(tb,k)

[0089] We wish to identify which set of smoothing gains maps a corresponding block of delayed audio (in delay line 102) to microphone signal M(k). The delay block (in line 102) is referred to as b best The corresponding block index of (t) is used as a rough estimate of the delay. In order to efficiently and reliably determine the rough delay estimate, subsystem 111 preferably calculates the error, the predicted spectrum and the power estimate of the actual microphone signal:

[0090]

[0091]

[0092]

[0093] The spectrum matching goodness factor can be defined as:

[0094]

[0095] This value is always in the range of 0 to 0.5. For each value of time t, subsystem 111 preferably tracks four values ​​of block index b that correspond to the four minimum values ​​of Q(t,b).

[0096] The goodness factor Q(t,b) is used to help determine which smoothing gains are best mapped to M(t,k). The smaller the goodness factor, the better the mapping. Therefore, the system identifies the block index b (of the block in delay line 102) that corresponds to the minimum value of Q(t,b). For a given time t, this is denoted b best (t). This block index is b best (t) provides a rough estimate of the time delay and is the result of the first (coarse) stage of time delay estimation performed by subsystem 93 as described above. The rough estimate of the time delay is provided to subsystems 106 and 107.

[0097] Preferably, the subsystem 111 determines the block index b best (t) Thereafter, subsystem 111 performs a thresholding test to determine the value corresponding to the best The smoothing gain H of the block (t) s (t,b best (t), k) should be considered as a gain candidate set for computing an optimized estimate of the time delay (i.e., for updating a previously determined optimized estimate of the time delay). If the test determines that all thresholding conditions are met, then a gain H is determined from it s (t,b best ,k) is considered a “good” (correct) block, and (in subsystems 105, 106, and 107) uses the value b best (t) and gain H s (t,b best , k) updating a previously determined optimized estimate of the delay (eg, determining a new optimized estimate L med (t)). If at least one of the thresholding conditions is not met, the previously determined optimized estimate of the delay is not updated. If the test indicates that the selected playback block (with index b best (t)) and its associated mapping (i.e., H s (t,b best (t), k)) is most likely the correct block that best maps to the microphone block M(t, k), then the previously determined optimized estimate of the delay is updated (e.g., as described below). After the tuning process, we have determined that it is preferable to apply three thresholding tests to determine whether the following three thresholding conditions are met:

[0098] 1)Q(t,b best (t)) < 0.4. This indicates that the gain H of the block s (t,b best ,k) provides a good mapping between M(t,k) and the playback data. In an alternative embodiment, a threshold other than 0.4 is used as the threshold (as mentioned above, Q(t,b best(t)) always has a value in the range of 0 to 0.5);

[0099] 2) where b 4th_best Denotes the block index b corresponding to the 4th smallest Q(t,b). As mentioned above, for each value of time t, the system tracks the four values ​​of block index b that correspond to the four smallest values ​​of Q(t,b), and can therefore determine the value of b for each time t. 4th_best In an alternative embodiment, some threshold other than 0.4 is used as the threshold. If a sinusoidal input is played through a loudspeaker, we have found that many playback blocks map well to M(t,k). Considering this and other similar cases, the second condition mentioned ensures that the selected mapping (corresponding to b best The mapping of Q(t,b) is a much better mapping than any other block index b. This ensures that the minimum goodness factor is very small compared to any other goodness factor. It is reasonable to expect the second and third minimum values ​​of the goodness factor Q(t,b) to be similar to the minimum value of the goodness factor, since these may correspond to neighboring blocks. However, the fourth minimum goodness factor Q(t,b) should be relatively large compared to the minimum value, and in these cases, H s (t,b best (t),k) is likely to be the correct mapping; and

[0100] 3) in is a control parameter (e.g. due to tuning, its value can be chosen based on the system and the expected use case). If P Mest (The estimated signal M est If the power estimate of the above power is too low, then the playback signal may be too small to be used to reliably and accurately update the delay estimate. Conversely, if the power of the estimated signal is high (e.g., above the threshold), then H s (t,b best (t),k) may be the correct mapping.

[0101] If the three thresholding conditions indicated above are met, the parameter ζ(t) is set equal to 1. In this case, the system estimates the value b based on a rough estimate. best (t) and gain H s (t,b best (t), k) to update (e.g., as described below) a previously determined optimized (sample accurate) delay estimate. Otherwise, parameter ζ(t) is set to have a value of 0. In this case, the previously determined optimized delay estimate (e.g., as described below) is used as the current optimized delay estimate L med (t).

[0102] Next, the optimized delay estimate L is determined. med (t) details of an example embodiment of Figure 2 Executed in subsystems 105, 106 and 107.

[0103] A typical analysis modulation through a decimated DFT filter bank has the following form:

[0104]

[0105] where α and β are constants, K is the number of frequency bands, M is the decimation factor or "span" of the filter bank, N is the length of the filter, and p(n) is the coefficient of the filter. A key aspect of some embodiments of the present invention is the recognition that the calculated gain coefficient H for mapping one block of complex frequency-domain audio data to another block is s (t, b, k) can also be viewed as an approximation of the transformed coefficients of the impulse response for which the corresponding operation has been performed in the time domain, assuming a reasonable implementation of each time-to-frequency transform filter (e.g., STFT or NPR DFT filter bank) used to generate the frequency domain data from which the delay is estimated. If the gain H is determined s (t,b best (t), k) is likely to provide a good mapping between the two audio data streams (e.g., by applying the three thresholding tests described herein), the system can then process the identified gain values ​​(H) with an inverse transform of the form s (t,b best (t), k), which corresponds to the value G(t, k) in the formula to calculate a new instantaneous delay estimate (used to update the previously determined instantaneous delay estimate):

[0106]

[0107] The calculation is also performed by identifying the location of the peak, ie the maximum value among the values ​​g(t,n) at time t.

[0108] This step of determining a new instantaneous delay estimate works well even when many values ​​of G(t,k) are zero, as they usually are due to the data reduction steps included in typical embodiments (e.g., in Figure 2 The same occurs as is performed in blocks 103 and 103A of the present embodiment, as long as the selected frequency bands are selected such that they are non-harmonically related (as described below).

[0109] Therefore, the system of the present invention ( Figure 2 A typical implementation of the subsystem 105 of FIG. 104 identifies a delay time b. best (t) The gain H of the delay block s (t,b bestThe peak value of the inverse transformed version of (t), k) (the "arg max" term in the following equation) is identified in a manner similar to that typically performed in correlation-based delay detectors. In subsystem 106, the delay time b is best (t) is added to this peak value to determine the optimized delay estimate L(t), which is the coarse delay estimate b best An optimized version of (t), such as in the following formula:

[0110]

[0111] where M is the decimation factor of the filter bank and K is the number of complex subbands in the filter bank. The summation over k is the result of applying the formula for the inverse complex modulated filter bank to H s _ in the estimated gain map data (many values ​​of k are not necessarily evaluated because based on data simplification, H s will be zero). The value of β must match the corresponding value of the analysis filter bank, and for a DFT modulated filter bank (e.g., STFT) this value is typically zero, but other implementations may have different values ​​that shift the center frequency of the frequency bin (e.g., 0.5). The parameter γ is some positive constant that controls how far away from the central peak the system appears.

[0112] The estimated value L(t) is provided to subsystem 107. When ζ(t) is 1 (determined by the thresholding test described above), subsystem 107 inserts L(t) into a delay line of length X (where X=40 in the exemplary embodiment, where this length has been determined using the tuning process, assuming an audio block size of 20 milliseconds). Subsystem 107 finds the median value of all the data in this delay line. This is denoted herein as L med This median value of ζ(t) is the final (optimized) estimate of the delay, which is reported to subsystem 109. When ζ(t) is zero, the previously generated median value is reported as the final estimate of the delay: med (t) = L med (t-1).

[0113] In typical operation, it is expected that Figure 2 The system estimated delay is fairly stable over time and over multiple iterations of the delay estimation subsystem 93. If this is not the case, then it is expected that the environmental and / or operating conditions of the system are undergoing changes; or the system is unable to accurately calculate the delay. In order to convey the latter situation to the user of the delay estimation subsystem, the subsystem 107 preferably generates and outputs (e.g., to the subsystem 109) at least one confidence metric (i.e., all or some of the values ​​C1(t), C2(t) or C(t)=C1(t)C2(t) mentioned below).

[0114] Next, an example of generating confidence metrics C1(t), C2(t), and C(t)=C1(t)C2(t) is described in more detail, which are heuristic confidence metrics because they are each determined using at least one heuristically determined parameter. As mentioned, subsystem 107 implements a delay line to determine a median value L of several recently determined values ​​L(t) med (t). In the example, the subsystem 107 exceeds the predetermined value N sim (For example, N sim = 10, which has been determined to be a suitable value by the tuning process in a typical use case) (each difference is a different value in the delay line and the median value L med (t)) counts the number of differences between the latest values. Make the value DV (similar to the median value L med The confidence metric C1(t) is divided by the total number of values ​​in the delay line, and the result is stored as a confidence metric C1(t), which corresponds to how many outliers there are in the delay line. If ζ(t) is zero, then the previously determined value of this confidence metric is taken: C1(t) = C1(t-1).

[0115] It is desirable for the system to indicate a high confidence level if the system measures the same delay for a period of time that is considered to be long. For example, in the case of a duplex communication device, the length of a Harvard sentence may be considered to be quite large. If the system sporadically measures different delays during this period, it is generally not desirable for the system to immediately indicate a loss of confidence. Preferably, the system will only indicate a loss of confidence if the system consistently (e.g., 80% of the time) estimates the same delay as the most recent estimate L. med Furthermore, when the operating condition changes from far-end only / two-way talk to near-end only, there is no playback audio data to estimate the delay, so the system should neither lose nor gain confidence in the calculated L med (t) confidence level.

[0116] To achieve all of these situations, subsystem 107 generates (and outputs) a new confidence metric C2(t) whose value increases slowly over time when subsystem 107 determines that many of the measured delay values ​​are the same, and decreases rapidly when the measured delay values ​​are different. An example of metric C2(t) is provided below. It should be appreciated that other ways of defining metric C2(t) are possible. Assuming the system tracks the parameter ζ(t) defined above, an example of metric C2(t) is as follows:

[0117] If ζ(t) = 1, and if the distance value D is less than N sim , where the distance value D is the most recently determined value L med The difference between the most recently determined values ​​of X at (t) and L(t) (e.g., N sim=10, as in the example described above),

[0118] C2(t)=C2(t-1)+a(1-C2(t-1)), where in a typical embodiment, a=0.3.

[0119] Otherwise, if P Mest (t,b best (t))≤1e -5 and C1(t)≤0.899,

[0120] C2(t)=C2(t-1)

[0121] Otherwise, if C2(t-1)>0.98 and C1(t)>0.9,

[0122] C2(t)=0.98

[0123] Otherwise, if C2(t-1)>0.5,

[0124] C2(t)=C2(t-1)-a(1-C2(t-1)), where in a typical embodiment, a=0.03.

[0125] otherwise,

[0126] C2(t)=(1-a)C2(t-1), where in a typical embodiment, a=0.03.

[0127] In the example, C2(t) is defined so that it rises logarithmically when the indicator indicates that the system should be more trustworthy, with the logarithmic rate ensuring that C2(t) is bounded by 1. However, when the indicator indicates that the system is losing confidence, the metric indicates less confidence, decaying slowly logarithmically so that it does not indicate loss of confidence due to any episodic measurements. However, if C2(t) decreases to 0.5, we switch to exponential decay for two reasons: to bound C2(t) by zero; and because in the case where C2(t) reaches 0.5, the system may be in new operating conditions / environment, so it will quickly lose confidence in L med (t). In the example, an additional condition is included for the case where C2(t-1)>0.98 and C1(t)>0.9. This is because the logarithmic decay is very slow at the beginning, so the example strongly drives the loss of confidence by setting C2(t) to 0.98. We expect that there are other ways to achieve the goal of measuring C2(t), which can be achieved by the described example.

[0128] A third confidence metric that may be generated (and output) by subsystem 107 is:

[0129] C(t)=C1(t)C2(t)

[0130] In some embodiments, subsystem 107 generates (and outputs) only the confidence metric C(t) or at least one of the metrics C1(t), C2(t) and C(t)=C1(t)C2(t) but not all. In other embodiments, subsystem 107 generates (and outputs) all metrics C1(t), C2(t) and C(t)=C1(t)C2(t).

[0131] Next, an example of data reduction implemented in some embodiments of the delay estimation method and system of the present invention is described in more detail (for example, in Figure 2 103A of the system). For example, data reduction may select only a small subset (e.g., 5%) of the frequency bands (with index k) of the audio data stream from which the delay is estimated, starting at a low value of index k (which is a prime number), and selecting the rest of the selected index k to be prime numbers. As mentioned previously, some embodiments of the inventive system operate only on a subset of the subbands of the audio data stream, i.e., only those for which the gain H is calculated. s (t,b best For some values ​​of index k that the system chooses to ignore (to improve performance), the system can s (t,b best (t), k) are set to zero.

[0132] As mentioned, the gain factor H that maps one composite audio data block to another block (according to the present invention, in the frequency domain) is s (t, b, k) is typically an approximation of the transformed coefficients of the impulse response performing the operation in the time domain. A selected subset of values ​​k should be determined to maximize (e.g., Figure 2 The inverse transform implemented in the subsystem 105 of s (t,b best (t), k), since gain values ​​are usually peaky looking data (which is how we want our impulse response to look). It can be shown that operating over a set of continuous values ​​of k is not optimal. Therefore, an exemplary embodiment of the delay estimation of the present invention operates over a selected subset of approximately 5% of the total number of transformed subbands, where those subbands have prime indexes, and where the first (lowest frequency) selected value is chosen to be at a level known to be detectable by the associated loudspeaker (e.g., Figure 2 The frequency reproduced by the system's speakers 91).

[0133] Figure 3is a graph showing the performance resulting from data reduction for a region of continuous values ​​of k relative to data reduction for a preferred choice of prime frequency band value k (a graph of system output indicated on the vertical axis versus time t indicated on the horizontal axis). The target impulse response (the pseudo impulse response with a peak at t=64) corresponds to the target impulse response that would be obtained by Figure 2 The curve labeled "Non-Linear Intervals of Selected (Non-Zeroed) Frequencies" is an example output of an inverse transform performed by subsystem 105 that operates on gains in only 5% of the full set of frequency bins (where the gains of the non-selected bins are zeroed), where the selected bins have prime frequency bin values ​​k. This curve has a peak that (ideally) aligns with the peak of the target impulse response.

[0134] The curve labeled "Linear Region of Zeroed Frequency" is an example output of an inverse transform implemented by subsystem 105 that operates on gains in only 5% of the full set of bins (where the gains of unselected bins are zeroed), where the selected windows encompass a region of consecutively numbered bin values ​​k. This curve does not have a peak that aligns with the peak of the target impulse response, indicating that the corresponding selection of windows is undesirable.

[0135] Example process

[0136] Figure 4 4 is a flow chart of an example process 400 for delay identification in the frequency domain. The process 400 may be performed by a system including one or more processors (e.g., Figure 2 System 200 or Figure 1 A typical implementation scheme of system 2) is performed.

[0137] The system receives (410) a first audio data stream and a second audio data stream (e.g., from Figure 2 According to an embodiment of the delay estimation method of the present invention, the system determines (420) a relative time delay (delay) between the first audio data stream and the second audio data stream in the frequency domain. The system also (e.g., Figure 2 The pre-processing subsystem 109 processes (430) the first audio data stream and the second audio data stream based on the relative delay.

[0138] The first audio data stream may originate from a first microphone (eg, Figure 1 Microphone 17 or Figure 2The second audio data stream may originate from a speaker tap in that the second audio stream is produced by a "tapped" speaker feed, e.g., when the speaker feed indicates audio data to be played out from the speaker. The determination operation 420 optionally includes calculating one or more confidence metrics (e.g., one or more of the heuristic confidence metrics described herein) indicating a confidence in determining a relative delay between the first audio data stream and the second audio data stream. The processing (430) of the first audio data stream and the second audio data stream may include correcting the relative delay in response to determining that the relative delay meets (e.g., exceeds) a threshold.

[0139] Example system architecture

[0140] Figure 5 This article is used as a reference Figure 1-4 Mobile device architecture for some embodiments of the described features and processes. Figure 5 The architecture 800 can be implemented in any electronic device, including but not limited to: desktop computers, consumer audio / visual (AV) equipment, radio broadcasting equipment, mobile devices (e.g., smartphones, tablet computers, laptops, wearable devices). In the illustrated example embodiment, the architecture 800 is for a smartphone and includes a processor 801, a peripheral device interface 802, an audio subsystem 803, a loudspeaker 804, a microphone 805, a sensor 806 (e.g., an accelerometer, a gyroscope, a barometer, a magnetometer, a camera), a position processor 807 (e.g., a GNSS receiver), a wireless communication subsystem 808 (e.g., Wi-Fi, Bluetooth, cellular), and an I / O subsystem 809 including a touch controller 810 and other input controllers 811, a touch surface 812, and other input / control devices 813. Other architectures with more or fewer components may also be used to implement the disclosed embodiments.

[0141] The memory interface 814 is coupled to the processor 801, the peripheral device interface 802, and the memory 815 (e.g., flash memory, RAM, and / or ROM). The memory 815 (non-transitory computer-readable medium) stores computer program instructions and data, including but not limited to: operating system instructions 816, communication instructions 817, GUI instructions 818, sensor processing instructions 819, mobile phone instructions 820, electronic message instructions 821, web browsing instructions 822, audio processing instructions 823, GNSS / navigation instructions 824, and application / data 825. The audio processing instructions 823 include instructions for executing the reference Figure 1-4 The audio processing instructions described herein (eg, instructions that, when executed by at least one of the processors 801, cause the at least one of the processors to perform an embodiment of the delay estimation method or steps thereof of the present invention).

[0142] Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) for buffering and routing data transmitted among the computers. Such networks may be constructed based on a variety of different network protocols, and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0143] One or more of the components, blocks, processes, or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logic components, and / or other features using any number of combinations of hardware, firmware, and / or embodied as data and / or instructions in various machine-readable or computer-readable media. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0144] Aspects of some embodiments of the invention include one or more of the following:

[0145] 1. A method for processing audio data to estimate a time delay between a first audio signal and a second audio signal, comprising:

[0146] (a) providing a first sequence of frequency domain data blocks M(t,k) indicating audio samples of the first audio signal and a second sequence of frequency domain data blocks P(t,k) indicating audio samples of the second audio signal, wherein t is an index indicating the time of each of the blocks and k is an index indicating a frequency band, and for each block P(t,k) in the second sequence, wherein t is an index indicating the time of each block, providing a delay block P(t,b,k), wherein b is an index indicating a block delay time, wherein each value of the index b is an integer number of block delay times of a corresponding one of the delay blocks delayed relative to the time t;

[0147] (b) For each block M(t,k), determine a rough estimate b of the delay at time t best (t), comprising determining an estimate M of said block M(t,k) when applied to each of said delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t, b, k) has the best spectral match with the block M(t, k), where the rough estimate b best(t) has an accuracy of approximately one of said block delay times; and

[0148] (c) Based on the rough estimate b best (t) and some of the gains determine an optimized estimate R(t) of the delay at time t, wherein the optimized estimate R(t) has an accuracy of approximately an audio sample time.

[0149] 2. The method according to technical solution 1, wherein the gain H(t,b,k) is the gain of each of the delay blocks P(t,b,k), wherein step (b) comprises determining a heuristic unreliability factor U(t,b,k) based on each frequency band for each of the delay blocks P(t,b,k), and wherein each of the unreliability factors U(t,b,k) is determined according to a set of statistical values, the set comprising: an average value H m (t, b, k), which is determined by averaging the two times according to the gain H(t, b, k); and the variance value H v (t, b, k), which is obtained by averaging the two times according to the gain H(t, b, k) and the average value H m (t,b,k) is determined.

[0150] 3. The method according to claim 1 or 2, wherein step (b) comprises the estimated value M for each value of the time t and index b. est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b).

[0151] 4. The method according to any one of technical solutions 1 to 3 further comprises:

[0152] (d) applying a thresholding test to determine whether a candidate optimized estimate of the time delay should be used to update a previously determined optimized estimate of the time delay R(t); and

[0153] (e) updating the previously determined optimized estimate R(t) of the time delay using the candidate optimized estimate only if the thresholding test determines that a thresholding condition is satisfied.

[0154] 5. The method according to claim 4, wherein step (d) comprises determining a value for the rough estimate b best (t), a set of smoothing gains H s (t,b best (t), whether k) should be considered as a gain candidate set for determining an updated optimized estimate of the delay.

[0155] 6. The method according to technical solution 4, wherein for a sequence of time t, multiple groups of gain H satisfying the threshold processing condition are used. s (t,b best (t), k), determining an optimized estimate R(t) of the time delay, and step (e) comprises identifying a median value in a set of X values ​​as the optimized estimate R(t) of the time delay, wherein X is an integer and the X values ​​comprise a most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the time delay.

[0156] 7. The method according to technical solution 4 further comprises determining a fourth best rough estimate b of the delay at time t 4thbest (t), and in which:

[0157] Step (b) comprises for each value of said time t and index b said estimated value M est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b), and

[0158] Step (d) comprises providing a rough estimate b best The goodness factor Q(t,b best ), for the fourth best rough estimate value b 4thbest The goodness factor Q(t,b 4thbest ) and for the rough estimate b best The estimated value of (t) is M est (t,b best ,k) Apply the threshold processing test.

[0159] 8. The method according to any one of technical solutions 1 to 7 further comprises:

[0160] At least one confidence metric is generated that indicates confidence in the accuracy of the optimized estimate R(t) of the time delay.

[0161] 9. A method according to technical solution 8, wherein the at least one confidence metric includes at least one or more heuristic confidence metrics.

[0162] 10. The method according to any one of technical solutions 1 to 9 further comprises:

[0163] Processing at least some of the frequency domain data indicative of audio samples of the first audio signal and the frequency domain data indicative of audio samples of the second audio signal includes performing time alignment by the optimized estimate R(t) based on the time delay.

[0164] 11. A method according to any one of technical solutions 1 to 10, wherein the first audio signal is a microphone output signal, and the second audio signal comes from a speaker tap.

[0165] 12. A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of technical solutions 1 to 11.

[0166] 13. A system for estimating a time delay between a first audio signal and a second audio signal, comprising:

[0167] at least one processor coupled and configured to receive or generate a first sequence of frequency domain data blocks M(t,k) indicating audio samples of the first audio signal and a second sequence of frequency domain data blocks P(t,k) indicating audio samples of the second audio signal, wherein t is an index representing the time of each of the blocks and k is an index representing a frequency band, and for each block P(t,k) in the second sequence, wherein t is an index representing the time of each block, providing a delay block P(t,b,k), wherein b is an index representing a block delay time, wherein each value of index b is an integer number of block delay times by which a corresponding one of the delay blocks is delayed relative to the time t, wherein the at least one processor is configured to:

[0168] For each block M(t,k), determine a rough estimate b of the delay at time t best (t), comprising determining an estimate M of said block M(t,k) when applied to each of said delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t, b, k) has the best spectral match with the block M(t, k), where the rough estimate b best (t) has an accuracy of approximately one of said block delay times; and

[0169] According to the rough estimate b best (t) and some of the gains determine an optimized estimate R(t) of the delay at time t, wherein the optimized estimate R(t) has an accuracy of approximately the audio sample time of the frequency domain data.

[0170] 14. A system according to technical solution 13, wherein the gain H(t, b, k) is the gain of each of the delay blocks P(t, b, k), and wherein the at least one processor is configured to:

[0171] Determine the rough estimate b best (t), comprising determining, for each of the delay blocks P(t,b,k), a heuristic unreliability factor U(t,b,k) based on each frequency band, wherein each of the unreliability factors U(t,b,k) is determined according to a set of statistical values, the set comprising: an average value H m (t, b, k), which is determined by averaging the two times according to the gain H(t, b, k); and the variance value H v (t, b, k), which is obtained by calculating the gain H(t, b, k) and the average value H at the two times. m (t,b,k) is determined.

[0172] 15. A system according to technical solution 13 or 14, wherein the at least one processor is configured to determine the rough estimate b best (t), comprising the estimated value M for each value of the time t and index b est (t, b, k), determine the goodness factor Q(t, b), and wherein determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b).

[0173] 16. A system according to any one of technical solutions 13 to 15, wherein the at least one processor is configured to:

[0174] applying a thresholding test to determine whether a candidate optimized estimate of the time delay should be used to update a previously determined optimized estimate of the time delay R(t); and

[0175] The previously determined optimized estimate R(t) of the time delay is updated using the candidate optimized estimate only if the thresholding test determines that a thresholding condition is satisfied.

[0176] 17. A system according to technical solution 16, wherein the at least one processor is configured to apply the threshold processing test, including by determining the rough estimate b best (t), a set of smoothing gains H s (t,b best (t), whether k) should be considered as a gain candidate set for determining an updated optimized estimate of the delay.

[0177] 18. A system according to technical solution 16, wherein the at least one processor is configured to, for a sequence of time t, process multiple sets of gain H according to the threshold processing condition. s (t,b best (t), k), determining an optimized estimate R(t) of the time delay, and updating the previously determined optimized estimate R(t) of the time delay using the candidate optimized estimate, including by identifying a median value in a set of X values ​​as a new optimized estimate R(t) of the time delay, where X is an integer and the X values ​​include a most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the time delay.

[0178] 19. A system according to any one of technical solutions 16 to 18, wherein the at least one processor is configured to:

[0179] Determine a fourth best rough estimate b of the delay at time t 4thbest (t);

[0180] Determine the rough estimate b best (t), comprising the estimated value M for each value of the time t and index b est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b); and

[0181] To the rough estimate b best The goodness factor Q(t,b best ), for the fourth best rough estimate value b 4thbest The goodness factor Q(t,b 4thbest ) and for the rough estimate b best The estimated value of (t) is M est (t,b best ,k) Apply the threshold processing test.

[0182] 20. A system according to any one of technical solutions 13 to 19, wherein the at least one processor is configured to generate at least one confidence metric indicating confidence in the accuracy of the optimized estimate R(t) of the time delay.

[0183] 21. A system according to technical solution 20, wherein the at least one confidence metric includes at least one or more heuristic confidence metrics.

[0184] 22. A system according to any one of technical solutions 13 to 21, wherein the at least one processor is configured to process at least some of the frequency domain data indicating the audio samples of the first audio signal and the frequency domain data indicating the audio samples of the second audio signal, including performing time alignment by the optimized estimate R(t) based on the time delay.

[0185] 23. A system according to any one of technical solutions 13 to 22, wherein the first audio signal is a microphone output signal, and the second audio signal comes from a speaker tap.

[0186] Aspects of the present invention include a system or device configured (e.g., programmed) to perform any embodiment of the method of the present invention, and a tangible computer-readable medium (e.g., disk) storing code for implementing any embodiment of the method of the present invention or its steps. For example, the system of the present invention can be or include a programmable general-purpose processor, digital signal processor, or microprocessor, which is programmed and / or otherwise configured with software or firmware to perform any of the various operations on data including embodiments of the method of the present invention or its steps. Such a general-purpose processor can be or include a computer system including an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform an embodiment of the method of the present invention (or its steps) in response to asserted data.

[0187] Some embodiments of the system of the present invention are implemented as a configurable (e.g., programmable) digital signal processor (DSP) or graphics processing unit (GPU), which is configured (e.g., programmed and otherwise configured) to perform the required processing on the audio signal, including performing an embodiment of the method of the present invention or its steps. Alternatively, an embodiment of the system of the present invention (or its elements) is implemented as a general-purpose processor (e.g., a personal computer (PC) or another computer system or microprocessor, which may include an input device and a memory), which is programmed and / or otherwise configured using software or firmware to perform any of the various operations including an embodiment of the method of the present invention. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or GPU or DSP configured (e.g., programmed) to perform an embodiment of the method of the present invention, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform an embodiment of the method of the present invention is typically coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0188] Another aspect of the present invention is a computer-readable medium (e.g., a disk or another tangible storage medium) storing code for executing any embodiment of the method of the present invention or its steps (e.g., code that can be executed to perform any embodiment of the method of the present invention or its steps).

[0189] Although specific embodiments of the invention and applications of the invention have been described herein, it will be apparent to those skilled in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the invention as described and claimed herein. It should be understood that although certain forms of the invention have been shown and described, the invention is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method for processing audio data to estimate a time delay between a first audio signal and a second audio signal, comprising: (a) providing a first sequence of frequency domain data blocks M(t,k) indicating audio samples of the first audio signal and a second sequence of frequency domain data blocks P(t,k) indicating audio samples of the second audio signal, wherein t is an index indicating the time of each of the blocks and k is an index indicating a frequency band, and for each block P(t,k) in the second sequence, wherein t is an index indicating the time of each block, providing a delay block P(t,b,k), wherein b is an index indicating a block delay time, wherein each value of the index b is an integer number of block delay times of a corresponding one of the delay blocks delayed relative to the time t; (b) For each block M(t,k), determine a rough estimate b of the delay at time t best (t), comprising determining an estimate M of said block M(t,k) when applied to each of said delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t, b, k) has the best spectral match with the block M(t, k), where the rough estimate b best (t) has an accuracy of approximately one of said block delay times; as well as (c) Based on the rough estimate b best (t) and one or more of the gains determine an optimized estimate R(t) of the delay at time t, wherein the optimized estimate R(t) has an accuracy of approximately an audio sample time, wherein the gain H(t,b,k) is the gain of each of the delay blocks P(t,b,k), wherein step (b) comprises determining, for each of the delay blocks P(t,b,k), a heuristic unreliability factor U(t,b,k) on a per-band basis, and wherein each of the unreliability factors U(t,b,k) is determined based on a set of statistical values, the set comprising: an average value H m (t, b, k), which is determined by averaging the two times according to the gain H(t, b, k); and the variance value H v (t, b, k), which is obtained by averaging the two times according to the gain H(t, b, k) and the average value H m (t,b,k) is determined.

2. The method of claim 1, wherein step (b) comprises: for each value of the time t and the index b, the estimated value M est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b).

3. The method according to claim 1, further comprising: (d) applying a thresholding test to determine whether a candidate optimized estimate of the time delay should be used to update a previously determined optimized estimate of the time delay R(t); and (e) updating the previously determined optimized estimate R(t) of the time delay using the candidate optimized estimate only if the thresholding test determines that a thresholding condition is satisfied.

4. The method according to claim 3, wherein step (d) comprises determining a value for the rough estimate b best (t), a set of smoothing gains H s (t,b best (t), whether k) should be considered as a gain candidate set for determining an updated optimized estimate of the delay.

5. The method according to claim 4, wherein for a sequence of time t, a plurality of sets of gain H satisfying the threshold processing condition are processed. s (t,b best (t), k), determining an optimized estimate R(t) of the time delay, and step (e) comprises identifying a median value in a set of X values ​​as the optimized estimate R(t) of the time delay, wherein X is an integer and the X values ​​comprise a most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the time delay.

6. The method of claim 3, further comprising determining a fourth best rough estimate b of the delay at time t 4thbest (t), and in which: Step (b) comprises for each value of said time t and index b said estimated value M est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b), and Step (d) comprises providing a rough estimate b best The goodness factor Q(t,b best ), for the fourth best rough estimate value b 4thbest The goodness factor Q(t,b 4thbest ) and for the rough estimate b best The estimated value of (t) is M est (t,b best ,k) Apply the threshold processing test.

7. The method according to claim 1, further comprising: At least one confidence metric is generated that indicates confidence in the accuracy of the optimized estimate R(t) of the time delay. The method of claim 7 , wherein the at least one confidence metric comprises at least one or more heuristic confidence metrics.

9. The method according to claim 1, further comprising: Processing one or more blocks of the frequency domain data indicative of audio samples of the first audio signal and the frequency domain data indicative of audio samples of the second audio signal includes performing time alignment by the optimized estimate R(t) based on the time delay.

10. The method of claim 9, wherein the processing comprises performing echo cancellation.

11. The method of claim 1, wherein the first audio signal is a microphone output signal and the second audio signal originates from a speaker tap.

12. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of claim 1.

13. A system for estimating a time delay between a first audio signal and a second audio signal, comprising: at least one processor coupled and configured to receive or generate a first sequence of frequency domain data blocks M(t,k) indicating audio samples of the first audio signal and a second sequence of frequency domain data blocks P(t,k) indicating audio samples of the second audio signal, wherein t is an index representing the time of each of the blocks and k is an index representing a frequency band, and for each block P(t,k) in the second sequence, wherein t is an index representing the time of each block, providing a delay block P(t,b,k), wherein b is an index representing a block delay time, wherein each value of index b is an integer number of block delay times by which a corresponding one of the delay blocks is delayed relative to the time t, wherein the at least one processor is configured to: For each block M(t,k), determine a rough estimate b of the delay at time t best (t), comprising determining an estimate M of said block M(t,k) when applied to each of said delayed blocks P(t,b,k) est (t,b,k) and identify the estimated value M est (t, b, k) has the best spectral match with the block M(t, k), where the rough estimate b best (t) has an accuracy of approximately one of said block delay times; as well as According to the rough estimate b best (t) and one or more of the gains determine an optimized estimate R(t) of the delay at time t, wherein the optimized estimate R(t) has an accuracy of approximately an audio sample time, wherein the gain H(t,b,k) is the gain of each of the delay blocks P(t,b,k), and wherein the at least one processor is configured to: Determine the rough estimate b best (t), comprising determining, for each of said delay blocks P(t,b,k), a heuristic unreliability factor U(t,b,k) on a per-band basis, Each of the unreliability factors U(t, b, k) is determined according to a set of statistical values, the set comprising: an average value H m (t, b, k), which is determined by averaging the two times according to the gain H(t, b, k); and the variance value H v (t, b, k), which is obtained by averaging the two times according to the gain H(t, b, k) and the average value H m (t,b,k) is determined.

14. The system of claim 13, wherein the at least one processor is configured to determine the rough estimate b best (t), comprising the estimated value M for each value of the time t and index b est (t, b, k), determine the goodness factor Q(t, b), and wherein determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b).

15. The system of claim 13, wherein the at least one processor is configured to: applying a thresholding test to determine whether a candidate optimized estimate of the time delay should be used to update a previously determined optimized estimate of the time delay R(t); and The previously determined optimized estimate R(t) of the time delay is updated using the candidate optimized estimate only if the thresholding test determines that a thresholding condition is satisfied.

16. The system of claim 15, wherein the at least one processor is configured to apply the thresholding test, including by determining a threshold for the rough estimate b. best (t), a set of smoothing gains H s (t,b best (t), whether k) should be considered as a gain candidate set for determining an updated optimized estimate of the delay.

17. The system according to claim 16, wherein the at least one processor is configured to process the plurality of sets of gains H according to the threshold processing condition for a sequence of time t. s (t,b best (t), k), determining an optimized estimate R(t) of the time delay, and updating the previously determined optimized estimate R(t) of the time delay using the candidate optimized estimate, including by identifying a median value in a set of X values ​​as a new optimized estimate R(t) of the time delay, where X is an integer and the X values ​​include a most recently determined candidate optimized estimate and a set of X-1 previously determined optimized estimates of the time delay.

18. The system of claim 15, wherein the at least one processor is configured to: Determine a fourth best rough estimate b of the delay at time t 4thbest (t); Determine the rough estimate b best (t), comprising the estimated value M for each value of the time t and index b est (t, b, k), determine the goodness factor Q(t, b), and determine the rough estimate b best (t) comprises selecting one of the goodness factors Q(t,b); and To the rough estimate b best The goodness factor Q(t,b best ), for the fourth best rough estimate value b 4thbest The goodness factor Q(t,b 4thbest ) and for the rough estimate b best The estimated value of (t) is M est (t,b best ,k) Apply the threshold processing test.

19. The system of claim 13, wherein the at least one processor is configured to: At least one confidence metric is generated that indicates confidence in the accuracy of the optimized estimate R(t) of the time delay.

20. The system of claim 19, wherein the at least one confidence metric comprises at least one or more heuristic confidence metrics.

21. The system of claim 13, wherein the at least one processor is configured to: Processing one or more blocks of the frequency domain data indicative of audio samples of the first audio signal and the frequency domain data indicative of audio samples of the second audio signal includes performing time alignment by the optimized estimate R(t) based on the time delay.

22. The system of claim 21, wherein the at least one processor is configured to implement a discrete Fourier transform (DFT) modulated filter bank to perform echo cancellation.

23. The system of claim 13, wherein the first audio signal is a microphone output signal and the second audio signal originates from a speaker tap.

Images