Adaptive step-size control of filters for echo cancellation

Abstract

In some embodiments, an echo cancellation method includes adaptation of at least one predictive filter, wherein the adaptation step size is controlled by using gradient descent on a set of filter coefficients of the filter, wherein the control of the adaptation step size is based at least in part on the direction of the adaptation and the predictability of the adaptation gradient (e.g., a gradient vector). Other aspects of embodiments of the present invention include systems, methods, and computer program products for controlling the adaptation step size of adaptive (e.g., low complexity adaptive) echo cancellation. In some embodiments, the adaptive step size control is based on normalized, scaled, or smoothed adaptive gradients including normalized.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 120,408, filed on December 2, 2020; U.S. Provisional Patent Application No. 62 / 990,870, filed on March 17, 2020; and U.S. Provisional Patent Application No. 62 / 949,598, filed on December 18, 2019, which are incorporated herein by reference. Field of the Invention

[0003] The present disclosure relates generally to audio signal processing (e.g., echo cancellation of audio signals). Some embodiments relate to performing echo cancellation using predictive filter adaptation, wherein the adaptation step size (e.g., the difference between consecutive estimates of a set of predictive filter coefficients) is controlled (e.g., to robustly and efficiently achieve echo cancellation). Background Art

[0004] The expression "echo cancellation" is used herein to refer to the suppression, elimination, or other management of the echo content of an audio signal.

[0005] Many commercially important audio signal processing applications, such as duplex communications and room noise compensation for consumer devices, benefit from echo cancellation. Echo management is a key aspect in any audio signal processing technology that requires duplex playback and capture, including voice communication technologies as well as consumer playback devices with voice assistants.

[0006] A typical implementation of echo cancellation involves the adaptation of one or more prediction filters. The one or more prediction filters take as input a reference signal and output a set of values ​​that are as close as possible to the corresponding values ​​observed in the microphone signal (i.e., with the smallest distance to the corresponding values). The prediction is typically accomplished using either a single filter (or a set of M filters) that operates on time domain samples of a reference signal frame, or one or more filters, each operating on data values ​​represented in the frequency domain of a reference signal frame.

[0007] When frequency domain data is predicted using a set of M prediction filters, the length of each of these filters is only 1 / M of the length of a single time domain filter required to capture the same delay range. During adaptation, the coefficients of one or more prediction filters are typically adjusted by an adaptive mechanism to minimize the distance between the input and the output of the one or more prediction filters. A variety of adaptive mechanisms are known in the art (e.g., LMS (Least Mean Square), NLMS (Normalized Least Mean Square), and PNLMS (Proportional Normalized Least Mean Square) adaptive mechanisms are conventional).

[0008] As described above, an echo cancellation system may operate in the time domain on a time domain input signal. For many audio samples (e.g., tens of thousands of audio samples), implementing such a system may be very complex, especially when using long time domain correlation filters, and may not produce good results.

[0009] Alternatively, the echo cancellation system may operate in the frequency domain on a frequency transformed representation of each time domain input signal (i.e., rather than operating in the time domain). Such a system may operate on a set of complex-valued bandpass representations of each input signal (which may be obtained by applying an STFT or other complex-valued uniformly modulated filter bank to each input signal). For example, U.S. Patent Application Publication No. 2019 / 0156852, published on May 23, 2019, describes echo management (echo cancellation or echo suppression) that includes frequency-domain adaptation of a set of prediction filters.

[0010] During echo cancellation, the need to adapt a set of prediction filters (e.g., using a gradient descent adaptive filter method) under any of a variety of signal and environmental conditions (e.g., in the presence of various types of noise) increases the complexity of the adaptation process. Conventional methods for controlling the adaptation step size introduce uncertainty (in the sense that the adaptation may not converge when they are used, or may not converge reliably and quickly enough under certain conditions). It would be useful to perform echo cancellation (including adaptation of one or more prediction filters) using adaptive step size control such that the adaptation is robust (i.e., converges reliably and quickly enough under a wide range of signal and environmental conditions, including in the presence of various types of noise) and efficient.

[0011] Symbols and terminology

[0012] Throughout the disclosure, including the claims, statements about performing an operation “on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0013] Throughout the disclosure including the claims, the term "system" is used in a broad sense to refer to a device, a system or a subsystem. For example, a subsystem that implements echo cancellation may be referred to as an echo cancellation system, and a system including such a subsystem may also be referred to as an echo cancellation system.

[0014] Throughout the disclosure, including the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing on audio data, graphics processing units (GPUs) configured to perform processing on audio data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0015] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device is referred to as being coupled to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0016] Throughout the disclosure, including the claims, "audio data" means data indicative of sounds (e.g., speech) captured by at least one microphone, or data that is generated (e.g., synthesized) so that the data can be rendered for playback (by at least one speaker) as sounds (e.g., speech). For example, audio data can be generated to be used as a substitute for data indicative of sounds (e.g., speech) captured by at least one microphone. Summary of the invention

[0017] In some embodiments, the present invention is an echo cancellation method comprising adaptation of at least one predictive filter, wherein the adaptation step size is controlled by using gradient descent on a set of filter coefficients (i.e., one or more filter coefficients) of the filter (i.e., a set of filter coefficients of the filter that have been previously determined), wherein the control of the adaptation step size is based at least in part on the direction of the adaptation and the predictability of the adaptation gradient.

[0018] In gradient descent adaptation, each adaptation step starts from the previous (i.e., current) set of filter coefficients θ n-1 Determine an updated set of filter coefficients θ n Each adaptation step consists of subtracting an update term (σ n ):θ n =θ n-1 -σ n , where each update term is at least partially determined by a function f[θ n-1 ] Here, "adaptive gradient" means the gradient Or a scaled (e.g., scaled and normalized) version of the gradient.

[0019] In the filter coefficient set θ n When multiple coefficients are involved, the function f[θ n-1 ],gradient Set θ n and the update term σ n Each of can be described as a vector, with each element of each vector corresponding to one of the coefficients. Each adaptive step size is an element of the vector, θ n -θ n-1 =σ n , which corresponds to a filter coefficient (or if the filter coefficient set θ n consists of only one coefficient, then the adaptive step size is a scalar value θ n -θ n-1 =σ n ).

[0020] Typically, adaptation is controlled such that when the adaptation gradient is as expected (i.e., with high predictability), adaptation proceeds quickly (with a relatively large step size), and when the adaptation gradient is not as expected (i.e., with low predictability), adaptation proceeds slowly (with a relatively small step size). The adaptation gradient typically depends on the prediction error, and the prediction error is expected to decrease (in one direction) from one adaptation step to another. Thus, in a typical embodiment, when the prediction error decreases (in one direction) as expected (e.g., under conditions of unexpected noise in an environment where echo cancellation is performed), adaptation is controlled to proceed faster (with a larger step size) than when the prediction error does not decrease (in one direction) as expected.

[0021] In some embodiments, the adaptive gradient is normalized and also scaled by a time-dependent factor (e.g., a time-varying weight s[t] described below) to control the adaptive step size (or facilitate control of the adaptive step size) based on the predictability of the normalized, scaled adaptive gradient. Some embodiments implement smoothing of the normalized adaptive gradient to improve control of the adaptive step size based on the predictability of the smoothed adaptive gradient.

[0022] In a first class of embodiments, each adaptation step (which determines updated filter coefficients a[t+1,k] in response to filter coefficients a[t,k]) is:

[0023]

[0024] where “·” denotes multiplication, “k” is an index identifying a filter coefficient a[k] that is updated at a series of different times (where a[t,k] denotes the value of a[t,k] at time t), X[t] is a time-varying weight, 1 / N is a normalization factor, |e[t]| is the absolute value of the error e[t] at time t, is the adaptive gradient.

[0025] In a first class of embodiments, the time-varying weights X[t] typically increase the adaptation step size (adaptation speed) when the error is decreasing as expected, and typically decrease the adaptation speed when the error is not decreasing as expected (in one direction) (e.g., under unexpected noise conditions in an environment where echo cancellation is performed). This is in addition to the control provided by the normalization factor 1 / N, since normalization of the adaptation gradients typically achieves faster adaptation (with convergence) under expected conditions (e.g., low unexpected noise conditions, when the error is decreasing over time as expected) than would be achievable without normalization.

[0026] The second class of embodiments utilizes modified accelerated gradient (MGA) descent to achieve adaptation. In the second class of embodiments, each adaptation step (which determines updated filter coefficients a[t+1,n] in response to filter coefficients a[t,n]) is:

[0027] a[t+1,n]=a[t,n]–β[n]σ[t+1,n]

[0028] where "n" is an index identifying a filter coefficient a[n] that is updated at a series of different times (where a[t,n] represents the value of a[n] at time t), and where β[n] is a weight based on the time index. Optionally, the weighting based on the time index is omitted (i.e., each β[n] can have a value of 1). In a second class of embodiments, the update term σ[t+1,n] is:

[0029]

[0030] Where γ is the smoothing factor, μ is the factor, and 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared error at time t, is the adaptive gradient. MGA descent implements adaptive smoothing, with the smoothing factor γ controlling the amount of smoothing (i.e., γ=0 results in no smoothing), for example to compensate for unexpected or unpredictable noise conditions. Normalization of the adaptive gradient typically enables faster adaptation (with convergence) than would be achievable without normalization under expected conditions (e.g., low unexpected noise conditions, when the error decreases over time as expected). Thus, normalization avoids too slow adaptation under normal or expected conditions (i.e., low noise conditions, where the prediction error decreases over time as expected to approach a minimum).

[0031] In some embodiments of the present invention, time-index-based weighting is used. For example, time-index-based weighting can be implemented by weight β[n] as in the second class of embodiments, or by weight μ[k], where in the first class of embodiments, X[t] is implemented as X[t]=μ[k]s[t], where s[t] is a time-varying weight. For example, in the case where each coefficient being updated belongs to a filter identified by the value of filter tap index l (determined using a filter bank), the weight μ(k) can depend on the filter tap index l of the filter including the coefficient being adjusted (identified by index k).

[0032] Nesterov accelerated gradient (NAG) adaptation with adaptive gradient normalization can achieve fast convergence under expected echo cancellation conditions (e.g., under normal or expected low noise conditions) while having adequate convergence under other conditions (e.g., under high, unexpected noise conditions). NAG adaptation by itself (i.e., without normalization) is often too slow under many operating conditions of the echo canceller. Normalizing the adaptive gradient (in gradient adaptation in addition to NAG adaptation) by itself can provide fast convergence at the expense of being less accurate when the adaptation is close to the target (e.g., under unexpected noise conditions).

[0033] According to typical embodiments, adaptation of prediction filter coefficients during echo cancellation can be controlled to be not only computationally efficient but also robust, in the sense that the adaptation converges reliably and sufficiently quickly, under a wide range of signal and environmental conditions (e.g., in the presence of various types and amounts of noise).

[0034] Aspects of the invention include systems configured (e.g., programmed) to perform any embodiment of the method of the invention or its steps, and tangible, non-transitory computer-readable media (e.g., disks or other tangible storage media) that implement non-transitory storage of data, the media storing code for performing (e.g., executable to implement) any embodiment of the method of the invention or its steps. For example, an embodiment of the system of the invention may be or include a programmable general-purpose processor, a digital signal processor, a GPU, or a microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the method of the invention or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform an embodiment of the method of the invention (or its steps) in response to data asserted thereto. Some embodiments of the system of the invention may be (or are) implemented as a cloud service (e.g., elements of the system are located in different locations, and data transmission is performed between these locations, such as over the Internet). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a block diagram of elements of an example echo cancellation system that implements predictive filter adaptation.

[0036] Figure 2 is a flow chart of an example echo cancellation process that includes predictive filter adaptation (eg, with adaptive step size control according to an embodiment of the present invention).

[0037] Figure 3 is a block diagram of an example echo cancellation system that may implement an acoustic echo cancellation algorithm with filter adaptation (eg, smoothing using normalized gradient vectors) according to an embodiment of the present invention.

[0038] Figure 4 is a flow chart of an example process for echo cancellation (eg, smoothing using normalized gradient vectors) according to an embodiment of the present invention.

[0039] Figure 5 is used to implement reference according to an embodiment Figure 1-4 Describes the characteristics and processes of mobile device architecture. DETAILED DESCRIPTION

[0040] An effective acoustic echo cancellation technique can utilize gradient descent on a set of filter coefficients to theoretically arrive at (i.e., determined by filter adaptation) an optimal set of echo cancellation filters (where the set includes one or more echo cancellation filters) that minimizes a prediction error (e.g., determined by a least squares method). In different embodiments of the present invention, different gradient descent methods (e.g., methods using normalization and / or smoothing of gradient vectors) are used to adapt at least one filter (i.e., stepwise adjust a series of states of the filter) to achieve a better approximation to an optimal version of the filter (e.g., with minimized prediction error). In one class of embodiments, gradient descent (e.g., smoothing using a normalized gradient vector) is used to control the filter adaptation step size.

[0041] Typical echo cancellation suffers from an adaptive filtering problem. One challenge with echo cancellation is that a microphone is capable of hearing multiple sound sources, but a typical echo cancellation system (used in or with a device that includes at least one microphone and at least one speaker) is intended to cancel only some of these sound sources. For example, in a conference call use case, the echo cancellation system may be designed to predict the linear components of the device speaker, but the microphone may (for example) receive speech from a person speaking near the microphone as well as nonlinearities produced by the device speaker. Given the signal being sent to the speaker (in the room or other environment), and the signal being received by the microphone (in the environment), echo cancellation must solve the following problem: How to form a filter (or set of filters) that will predict the signal in the microphone based on the signal being sent to the speaker? If the echo cancellation system can determine such a filter, then that filter can be used to subtract the predicted signal from the microphone signal to determine the remaining signal in the room (or other environment).

[0042] refer to Figure 1 , in an example echo cancellation system, m[t] is a microphone signal captured by a microphone, r[t] is a reference signal sent to a speaker, and e[t] is an error signal generated by subtracting a filtered version of the reference signal from the microphone signal m[t]. An element labeled "adaptive filter" can determine a filter a[t] having filter coefficients and apply the filter to the reference signal r[t] to produce a signal subtracted from the microphone signal m[t]. At each time in a series of different times t, the filter is adjusted (updated with an adaptive filter step size control that can be implemented according to an embodiment of the present invention) to minimize the error e[t]. The updated filter (determined for each time t) can then be used to filter the microphone signal (to suppress or eliminate the echo content of the microphone signal). The updated filter can be used until a new updated version thereof (at the next time in the time series) is determined.

[0043] An example of the error e[t] is as follows:

[0044]

[0045] If the filter is implemented in the time domain, the filter needs to contain many useful coefficients. Adapting such a large filter is computationally expensive and algorithmically difficult to produce fast convergence. It is generally preferred to use a set of M filters (where M is a number), each of which is a small filter used to filter a subset of data values ​​represented by a frequency domain segment (e.g., a frame) of a reference signal. Therefore, typical embodiments of the present invention utilize a filter bank, such as a short-time Fourier transform (STFT) or a near-perfect reconstruction DFT filter bank, to replace a large time domain filter of a specified type with several (e.g., multiple) smaller filters (each with a different index l) (i.e., by effectively decomposing the large filter into the several smaller filters), so that the filter adaptation problem is typically how to determine the optimal set of filter coefficients a for each index value l in the frequency domain (for each time t in the time series). l (Written as "a l [k]"):

[0046]

[0047] where l is the index of the component (the "l"th filter) of the filter bank. In other words, the output of the filter bank is a set of filters, each identified by a different value of the index l. The adaptation of these filters at each time t consists in minimizing the error e of each filter l [t], to determine the updated filter bank at time t.

[0048] If we assume that there is no other noise in the room (or other environment), we can use the magnitude of e[t] or each error e l [t] is regarded as minimizing the objective function, and in the initial filter coefficient set (for example, the initial coefficient set a l ) to find the best filter coefficient set at time t (e.g., the best coefficient set a l ). However, there are often other noise sources in the room (possibly including other people talking) to which you do not want any filter to adapt. It is often undesirable to try to create a filter that attempts to predict not only the desired content (e.g., the user's speech captured by the microphone) but also all other audio sources. Various techniques have been proposed for filter adaptation in echo cancellers that avoid trying to adapt to audio other than the desired content (e.g., the user's speech captured by the microphone).

[0049] Once the filter coefficients are properly adapted at time t, the error e[t] (or each error e l [t]) represents the unpredictable component of the loudspeaker signal plus the audio in the room (or other environment). It is desirable that the level of the unpredictable component be significantly lower than the loudspeaker signal itself, but the unpredictable component often still needs to be further suppressed using other mechanisms.

[0050] Next, refer to Figure 2 , describing an example of echo cancellation according to a class of embodiments of the present invention. Figure 2 is a flow chart of an example echo cancellation process 200 including adaptive filter step size control. Process 200 may be performed by a system (eg, an echo canceller) including one or more processors.

[0051] In process 200, an echo canceller receives (in step 210) an input signal from a microphone, and the echo canceller receives (in step 220) an output signal (speaker feed signal) to a speaker. Typically, the speaker and microphone are implemented in a single device. The echo canceller predicts (in step 230) a portion (i.e., content) of the input signal (signal captured by the microphone) that is caused by the speaker (i.e., produced by sound emitted by the speaker and captured by the microphone). The prediction (step 230) includes configuring (including initializing and adapting) an adaptive filter based on the input signal and the output signal. The configuration may include scaling (or otherwise controlling) an adaptive rate of the adaptive filter (e.g., based on at least one of an index of a filter tap or an energy of an error signal, as described below) according to an embodiment of the present invention. The echo canceller removes (in step 240) the portion (i.e., content) of the input signal caused by the speaker from the input signal.

[0052] In some implementations of process 200, step 230 includes adapting a set of filters (each filter including coefficients having different filter tap index values), and the adaptation rate is controlled (according to an embodiment of the invention) to be slower for increasing filter tap index values. In some implementations of process 200, the adaptation rate of at least one filter is controlled (according to an embodiment of the invention) to increase in response to a decrease in error signal energy, and to decrease in response to an increase in error signal energy. Typically, the adaptation rate is allowed to increase and decrease to respective limit values.

[0053] Filter adaptation according to some embodiments of the present invention uses gradient descent. Using gradient descent to construct (adapt) an adaptive filter relies on being able to calculate the partial derivatives of the error function for each filter coefficient. Then, during adaptation, the filter coefficients are moved (changed) by a value that depends on the partial derivatives, i.e.:

[0054]

[0055] Wherein, "k" represents the filter coefficient being adjusted (i.e., the "k"th filter coefficient is adjusted), and "μ" is the scaling factor. In some embodiments, there are multiple different filters (and are subject to adaptation), each filter consisting of coefficients corresponding to different filter bank taps, each of such coefficients being identified by a different value of the filter tap index "l".

[0056] If this factor μ is too large (for adaptation according to the above equation), the filter may not converge even for good inputs. If μ is too small, the filter will adapt very slowly. As the filter approaches the minimum of the error function (during adaptation), the partial derivatives become smaller, resulting in slower convergence. One known method to try to solve this problem is to use another dynamic weighting (normalization factor) during adaptation, for example, the square root quantity in the denominator of the following equation:

[0057]

[0058] In the above equation, the range of index "n" includes all values ​​of index k, so the sum is performed over all available values ​​of k (all adjusted filter coefficients). This equation determines the updated value of one of the filter coefficients (which has an index "k").

[0059] In the equation in the previous paragraph, μ becomes related to the maximum absolute value that a single coefficient may change in each adaptation iteration. This method works well until a signal is introduced at the microphone that is related to the audio being played back by the device (i.e., noise is introduced) (for example: a person is talking near the device while the speaker is also playing speech).

[0060] Next, two example embodiments of the inventive method are described, which address this limitation of the above-described adaptive method.

[0061] In both example embodiments, the filter adaptation step is performed (occurs) at time t+1, assuming that the filter has been adjusted (or initialized) at an earlier time t. Typically, the adaptation is performed multiple times (each adaptation starts at a different time). In these examples, the index "k" indicates which filter coefficient is being adjusted (i.e., these equations belong to the "kth" filter coefficient being adjusted). Typically, there are multiple different filters, each corresponding to a different filter bank tap identified by a different value of the index "l". The symbol a[t,k] represents the coefficient of a filter that has been adjusted (or initialized) at time t. Each ("kth") filter coefficient is adjusted in a manner described by the reference coefficient value a[t,k]. Typically, each filter that is adjusted includes only a small number of coefficients (which can be identified by different values ​​of the filter tap index "l"), so that it is stably constructed. In the first example embodiment, each filter that is adjusted consists of 8 coefficients, each coefficient corresponding to a different filter bank tap identified by a different one of the 8 values ​​of the index "l".

[0062] First example: Scaling μ based on filter tap index

[0063] The first example embodiment recognizes the fact that in any typical environment, the shape over time of each echo cancellation filter should decay (it is not desirable to perform or to perform echo cancellation in an environment where the echo strength increases over time). Rather than having all filter coefficients move at the same speed (during adaptation), coefficients closer to time zero are allowed to move faster than coefficients further away in time. Therefore, in order to handle the case where the microphone signal indicates other data (which might cause the partial derivatives defined below to attempt to drag the filter into a non-decaying shape during adaptation), a weighting factor (μ[k]) is introduced that penalizes attempts to construct a filter that does not decay.

[0064] In this example, the weighting factor μ in the above formula is replaced by a set of weighting factors μ[k]. This example assumes that each adjusted filter consists of 8 coefficients, each coefficient corresponding to a different filter bank tap identified by a different index value "l". Each factor μ[k] is associated with a coefficient identified by a different index value l (and is used for the adaptation of the coefficient).

[0065] In a typical implementation of the example embodiment, the echo canceller of the present invention operates using a filter bank that extracts the audio signal at 20 milliseconds. For each filter bank band, there are 8 complex tapped adaptive filters (each "tap" is identified by a different index value l), giving the canceller the ability to cancel approximately 160 milliseconds of echo. A suitable set of weighting factors μ[k] for these filters is:

[0066] For the filter coefficient with tap index l=0, μ[k]=0.004;

[0067] For the filter coefficient with tap index l=1, μ[k]=0.004;

[0068] For the filter coefficient with tap index l=2, μ[k]=0.002;

[0069] For the filter coefficient with tap index l=3, μ[k]=0.001;

[0070] For a filter coefficient with tap index l=4, μ[k]=0.0004;

[0071] For the filter coefficient with tap index l=5, μ[k]=0.0004;

[0072] For the filter coefficient with tap index l=6, μ[k]=0.0001;

[0073] For a filter coefficient with tap index l=7, μ[k]=0.0001.

[0074] In variations of the exemplary set of weighting factors μ[k], other values ​​of the weighting factors μ[k] are used. Typically, the weighting factors for filter taps with lower index values ​​l are greater than (or equal to) the weighting factors for filter taps with higher index values ​​l.

[0075] The weighting factor μ[k] may be applied in each filter adaptation step performed according to the second example embodiment described below. For example, in the following equation for the adaptation step of the second example embodiment, the weighting factor μ[k] is taken as indicated in the numerator (of the last term on the right side of the equation), multiplied by the factor s[t], and divided by the normalization factor (the square root quantity in the denominator of the last term on the right side of the equation). In a variation of this example, one or both of the factor s[t] and the normalization factor are omitted (i.e., replaced by the value "1").

[0076] Second Example: Dynamic Scaling of μ Based on the Energy of the Error Signal e[t]

[0077] The example embodiments are examples of gradient descent adaptation. A second example embodiment employs time-varying weights s[t] modified according to the direction and amount of prediction error movement. Typically, it also employs the weighting factors μ[k] described above, but in some cases these factors may be omitted (i.e., replaced by factors having a value of "1"). In the second example embodiment, the filter adaptation step (which determines the updated filter coefficients a[t+1, k] in response to the filter coefficients a[t, k]) is:

[0078]

[0079] where, in a typical implementation, s[t] is defined as:

[0080]

[0081] In the above equations, α, β, γ, and δ are configurable parameters, and the range of index "n" includes all values ​​of index k. Therefore, the sum in the denominator (i.e., the normalization factor) is over all coefficients that are adjusted (each coefficient is identified by a different value of k). More specifically, the sum is over the partial derivatives of the squared error for all index values ​​k. Each adjusted filter coefficient is identified by the value of index "k", and different values ​​of the factor "μ[k]" generally correspond to different filter bank taps (with filter tap index l).

[0082] When there is no audio stimulus (captured by the microphone) except for the audio stimulus produced by the speaker of the device, it is expected that the error e[t] should decrease most of the time (during a series of filter adaptation steps) as the filter coefficients a of all filters move toward a result. Therefore, when the indicated condition is met (the absolute value of e[t] is less than the absolute value of e[t-1]), the parameter α in the expression of s[t] is preferably set to a value slightly greater than 1 to increase the adaptive step size. When such an audio stimulus is present, the opposite is expected: typically, the error increases over time. Therefore, the parameter β in the expression of s[t] is preferably set to a value slightly less than 1 to reduce the step size when the corresponding condition is met. The step size range is limited by the specific values ​​of the parameters γ and δ. In implementation, given the 8 example values ​​of μ[k] above, the values ​​of α, β, γ and δ can be 1.01, 0.99, 0.005 and 8.0, respectively.

[0083] In an exemplary embodiment, s[t] has a relatively large value when the absolute value of the error e[t] decreases (i.e., is less than the absolute value of the error e[t-1]). Larger values ​​of s[t] (and / or larger values ​​of μ[k]) tend to increase the speed of adaptation (i.e., increase the adaptation step size), while smaller values ​​of s[t] (and / or μ[k]) tend to decrease the speed of adaptation (i.e., decrease the adaptation step size). This has the effect of reducing the step size to zero when potential double talk occurs (e.g., when the error e[t] does not decrease over time), thus preventing the filter coefficients a from changing rapidly. When environmental conditions are good, exemplary embodiments allow the adaptation step size to increase so that the adaptation moves quickly (to improve adaptation time).

[0084] Figure 3is a block diagram of an example echo cancellation system that may implement an embodiment of the acoustic echo cancellation algorithm of the present invention (eg, an embodiment that performs filter adaptation using gradient descent adaptation (eg, smoothing using normalized gradient vectors)).

[0085] Figure 3 The system may be a communication device including a processing subsystem (at least one processor programmed or otherwise configured to implement an audio processing subsystem 111, a communication application 113, a media player 112, and a voice assistant 114) and physical device hardware coupled to the processing subsystem (including a speaker 101 and a microphone 102). Typically, the system includes a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform an embodiment of the method of the present invention.

[0086] The audio processing subsystem 111 (eg, implemented as an audio processing object) may be implemented (ie, Figure 3 At least one processor of the system is programmed to execute subsystem 111 to perform an embodiment of the echo cancellation method of the present invention. Subsystem 111 is configured to generate (e.g., implement a filter bank that generates) or receive frequency domain playback audio data indicating the audio content of a playback audio signal provided to speaker 101 (speaker feed, sometimes referred to herein as a "reference" signal), and frequency domain microphone data indicating the audio content of a microphone signal output from microphone 102.

[0087] Subsystem 103 of subsystem 111 (in Figure 3 1 (labeled as "AEC" in the figure) is an echo cancellation subsystem that is configured to perform echo cancellation (e.g., an embodiment of the acoustic echo cancellation algorithm of the present invention). Subsystem 111 is also implemented to (e.g., it includes a speech processing subsystem 104, which is implemented to) perform other audio processing on the output of echo cancellation subsystem 103.

[0088] Subsystem 111 may be implemented with Figure 3 A software plug-in for audio data interaction that exists within the system's processing subsystem.

[0089] exist Figure 3 In a typical implementation of , time domain reference audio data r[n] including reference signal samples provided to speaker 101 and time domain microphone audio data m[n] including microphone signal samples output from microphone 102 are provided to subsystem 111. The subsystem of echo cancellation subsystem (echo canceller) 103 (in Figure 3The subsystem labeled "prediction" in the example above implements a filter bank that performs a time-to-frequency transform on the data r[n], a time-to-frequency transform on the data m[n], and generates an initial set of prediction filters (each with a different index l). Each prediction filter includes an initial set of filter coefficients a l [k] The subsystem 103 is configured to determine an optimal set a of filter coefficients for each index value l in the frequency domain (for each time t in the time series) by performing adaptation on the initial set of filter coefficients for the index value l. l [k].

[0090] Echo cancellation is performed in response to a reference signal (a speaker feed indicating audio content to be played from speaker 101) and a microphone signal (indicative of audio content captured by microphone 102). The microphone signal may undesirably contain audio content emitted from speaker 101. Typically, the output of echo canceller 103 is an echo-managed version of the microphone audio, from which it is desirable to remove as much speaker audio as possible or practical. The output of echo canceller 103 is provided to communication application 113, and optionally also to voice assistant 114.

[0091] The echo cancellation process is typically implemented in a manner that includes attempting to estimate a filter (or each of a set of filters) that maps reference audio (the content of the reference signal) to microphone audio (the content of the microphone signal). More specifically, each filter is determined by an adaptive process in an effort to determine an adjusted filter that can filter audio data that indicates the audio content (reference audio) that has been sent to the speaker, and the adaptation attempts to determine a linear combination of values ​​that best estimates the microphone audio (a filtered version of the reference audio, sometimes referred to as an estimated echo). The microphone audio is then filtered using the adjusted (estimated) filter in an effort to subtract the estimated echo from the microphone audio.

[0092] A low-complexity solution for echo cancellation uses a gradient descent technique (e.g., an embodiment of the filter adaptation method of the present invention) to find out how to update (adapt) each prediction filter in a way that minimizes a cost function. The cost function is typically defined as the squared error between the estimated echo signal (the filtered version of the reference audio) and the microphone audio. Gradient descent typically assumes a linear relationship between the input and output audio, but this does not occur in actual devices due to the nonlinearity of the system and the presence of other noise sources, which prevents these techniques from producing good output. There are many ways to perform each filter update (i.e., the adaptation of the filter to produce an updated filter), and the update method can be selected to optimize different aspects of the canceller (e.g., the optimization takes into account how quickly the echo canceller finds a reasonable filter and / or how much echo the filter can reduce). Embodiments of the inventive method disclosed herein typically implement filter adaptation so that the filter adapts at a desired rate (e.g., fairly quickly) and robustly so that the adjusted filter can produce the desired amount of echo suppression.

[0093] Still reference Figure 3 , the reference audio r[n] played back via the speaker 101 is obtained from the mixer 105, which may receive audio from multiple sources. Echo cancellation is performed in response to the microphone audio m[n] from the microphone 102 and the reference audio r[n].

[0094] Subsystem 103 (the area surrounded by the dashed line in subsystem 111) is an echo canceller. As can be seen, the echo canceller introduces the microphone and reference audio into a "prediction" block, which creates filter coefficients by which the reference audio is filtered to produce p[n], which is the prediction signal. This signal is then subtracted from the microphone signal to produce the echo canceled output. Taken alone, the echo canceled signal may still not be suitable for voice communication and may require further "cleaning" to remove noise and echo components that cannot be removed by the canceller. In Figure 3 In a typical implementation of the system, such additional processing may be performed in block (speech processing subsystem) 104. The resulting output audio is then passed to the communication application 113 and / or voice assistant 114. Furthermore, if the application that desires the audio output is a voice assistant, the configuration of the system may benefit from operating in a different configuration than if the communication application receives the output audio.

[0095] Next, an exemplary embodiment of the echo cancellation method of the present invention is described, which uses a gradient descent filter adaptation method (which controls the adaptation step size) to achieve the adaptation of at least one prediction filter (e.g., a group of prediction filters). The exemplary embodiment can be composed of Figure 3 The echo cancellation subsystem 103 of the system or other embodiments of the system of the present invention is implemented.

[0096] Gradient descent adaptation takes a function f(θ) of a certain parameter vector θ (e.g., a parameter vector of prediction filter coefficients) and uses the gradient of the function with respect to one or more of the parameters (e.g., one or more filter coefficients) to adjust the current estimate of at least one (e.g., all) of the parameters to approach a certain minimum value. Although the parameter vector θ may consist of multiple parameters (e.g., in some embodiments of the present invention, it consists of multiple filter coefficients, each of which is a coefficient of a different prediction filter), in some cases it may consist of only one parameter (the filter coefficient). More specifically, although echo cancellation may include adaptation of a set of coefficients of a set of filters (e.g., as described above, each filter is identified by a different index value l), some descriptions of gradient descent embodiments herein clearly describe adaptation of only one coefficient of one such filter (e.g., at each time t of the time series, including minimizing the error e[t] of the coefficient), but the adaptation may include normalization by a factor determined from the multiple filter coefficients. In each case where multiple filter coefficients (e.g., a vector of coefficients of multiple prediction filters) are to be updated, each coefficient may be adjusted in the manner described herein.

[0097] In the gradient descent adaptation implemented in an acoustic echo canceller, the function f(θ) can be defined such that it is the square of the total error of subtracting the prediction signal (a filtered version of the speaker feed delivered to the speaker) from the microphone signal, where the parameters of the vector θ are the coefficients of the prediction filter (or set of prediction filters). The expression e is sometimes used. 2 [t] represents the total squared error between the microphone signal m[t] and the filtered version of the audio delivered to the speaker r[t], where a[t] is the prediction filter coefficient (applied to r[t] to determine the filtered version of r[t]). Although the error function e 2 [t] is a function of time, but is sometimes called e 2 (θ), since θ may be the vector of filter coefficients a[t] at time t.

[0098] When performing some implementations of gradient descent to adapt a set of prediction filter coefficients, each adaptation step includes subtracting the gradient (partial derivative) of the function f(θ) with respect to the vector θ, or subtracting the gradient of a modified (e.g., scaled, weighted, and / or smoothed) version of the gradient of the function f(θ) in an effort to "step" to zero error. In other words, (prediction filter coefficients θ n Each step of gradient descent adaptation of the current set of ) can determine a set of updated filter coefficients θ as follows n+1 :

[0099]

[0100] where μ is a factor (e.g., a weighting factor or a weighting and normalization factor). The function f(θ n ) and partial derivatives Each of them is also a vector, with the vector θ of filter coefficients n The same number of elements. In the formula, the index "n" represents a time (a time in the update time series).

[0101] Various methods have been proposed to control the adaptive step size θ in gradient descent filter adaptation. n+1 -θ n (Depending on the range of index n, this could alternatively be written as θ n -θ n-1 ).

[0102] We next describe three categories of these methods. In each example method, the filter coefficients are obtained by subtracting a vector (σ) from the previous (i.e., current) set (vector) of filter coefficients. n ), according to the current vector θ of filter coefficients n-1 To determine each update vector θ of the filter coefficients n :

[0103] θ n =θ n-1 -σ n .

[0104] Three examples of gradient descent filter adaptation in how to define the vector σ n There is a difference.

[0105] Determine the vector σ n Three examples are as follows:

[0106] 1.

[0107] Where “·” represents multiplication, μ is a factor, and f[θ n-1 ] is θ n-1 The function of is f[θ n-1 ] About θ n-1 The partial derivative of

[0108] 2.

[0109] Where “·” represents multiplication, μ is a factor, and f[θ n-1 ] is θ n-1 The function of is f[θ n-1 ] About θ n-1 The partial derivative of θ.n-1 is a vector (consisting of one or more filter coefficients), so is a vector consisting of elements, where each element is f[θ n-1 ] is the partial derivative of a filter coefficient with respect to different coefficients. The quantity in the denominator is a normalization factor (e.g., The square root of the sum of (at all index values ​​x), where each θ x-1 is the vector θ n-1 one of the filter coefficients of , and each different value of the index x identifies a different one of the filter coefficients); and

[0110] 3.

[0111] where “·” represents multiplication, γ and μ are factors, and f[θ n-1 ] is θ n-1 The function of is f[θ n-1 -γσ n-1 ] About θ n-1 The partial derivative of .

[0112] As mentioned above, in each gradient descent adaptation step, θ n =θ n-1 -σ n , the next set of filter coefficients θ n (i.e., the predicted filter coefficient or coefficients for time “n”) are obtained by taking the current set of filter coefficients θ n-1 Subtract the vector σ from n To obtain.

[0113] Determine σ n The first method (numbered “1” above) is the classic stochastic gradient descent method, in which each gradient is scaled by a factor μ. Once the error function f[θ n-1 ] starts to approach zero during the adaptation process, and the parameters (filter coefficients θ n ) by gradually moving in smaller and smaller amounts. However, this approach is known to adapt slowly. For cases where the system is dynamic (e.g. when performing adaptation to update the prediction filter of an echo canceller), it usually performs poorly and never achieves good results due to noise in the optimization path.

[0114] Determine σ n The second method (numbered "2" above) converts the gradient vector Normalize and scale the normalized gradient vector by a factor μ. In this case, the factor μ provides a way to compromise between adaptation speed and adaptation accuracy. Care should be taken to limit the value of μ to ensure that the system remains stable, while not choosing a value that is too small so that the system cannot adapt well.

[0115] Determine σ n The third method (numbered "3" above) is called the Nesterov Accelerated Gradient method. This method includes the additional term γσ n-1 And use the gradient vector Replace the gradient vector Instead of finding the gradient (derivative parameter) based on its current value, this approach assumes that the derivative has continued to move forward in its current direction for some distance (this is because the derivative is effectively smoothed, which can be seen from σ n Its previous value σ n-1 The derivative is determined by looking at the dependence of .

[0116] Next, an embodiment of the filter adaptation method of the present invention (Modified Gradient Acceleration or "MGA" embodiment) is described, which implements a modification of the Nesterov Accelerated Gradient (NAG) method to optimize the prediction filter coefficients θ to be optimized. n-1 (i.e., for the prediction filter coefficients θ n-1 This embodiment is the above method for selecting σ n A modified version of the third method where the gradient vector Rather than just scaling by a rate factor μ, it scales by the amount μ / N, where μ is the rate factor and 1 / N is a normalization factor. Next, an MGA embodiment is described in more detail.

[0117] In the MGA embodiment, the error signal is defined as:

[0118] e[t]=m[t]–p[t]

[0119] where p[t] is the prediction signal (i.e., the signal that predicts the microphone signal m[t] from the loudspeaker signal). In some filter adaptation implementations, the prediction signal is defined (as above) as:

[0120]

[0121] where a[t,k] are the prediction filter coefficients and r[t] is the speaker feed sent to the speaker.

[0122] In the exemplary MGA embodiment now described, this definition is modified to define the error signal p[t] as:

[0123]

[0124] where a[t,k] are the prediction filter coefficients, r[t] is the speaker feed sent to the speaker, and σ is the current set (vector) of prediction filter coefficients (denoted as θ above) n-1 ) to determine the update vector for the prediction filter coefficients (denoted as θ above). n ), where γ is the smoothing factor (i.e., there is no smoothing in the case of γ = 0).

[0125] In the general description of the Nesterov Accelerated Gradient (NAG) adaptation technique above, the update vector σ is defined using the index “n” to represent the update time. n , so that σ n represents a vector of update times (where the vector has a component for each filter coefficient that is updated at that time), and σ n+1 A vector representing the next update time (where the vector has a component for each filter coefficient that is updated at the next update time). To complete the description of the exemplary MGA embodiment of the present invention, for convenience, a different notation "σ[t,n]" is used to represent the elements of each update vector. More specifically, the update vector (at time t) is composed of multiple elements, and each element of the update vector at time t is "σ[t,n]" in the new notation, where the index "n" distinguishes elements of the same update vector. In the new notation, an update vector with elements σ[t,n] corresponds to the update vector σ defined above. n , where “σ n The index "n" in " represents the time.

[0126] Using the new notation, assume that at time t, a set (vector) of prediction filter coefficients (each coefficient is identified by a different index value n) is adjusted. Each prediction filter coefficient is "a[t,n]". Therefore, in the new notation, σ[t,n] is an element of the update vector used to update the filter coefficients "a[t,n]".

[0127] Using the new definition of p[t]

[0128]

[0129] Error term e 2 [t] is:

[0130] e 2 [t]=(m[t]-p[t]) 2 .

[0131] For simplicity, in the following discussion, each filter coefficient a[t,n] is written as “a[n]”. Thus, is the squared error e at time t 2 The partial derivative of a[n] with respect to the coefficient a[n] at time t. The partial derivative is:

[0132]

[0133] where "r[t]" represents the loudspeaker feed filtered by the prediction filter and "m[t]" represents the microphone signal.

[0134] For convenience, we define the normalized quantity f[t] as:

[0135]

[0136] In the definition of the normalized quantity f[t], the sum is the partial derivative with respect to all prediction filter coefficients a[n] (i.e., the sum index k covers all possible values ​​of the index "n" identifying the filter coefficient a[n]). Although the sum notation allows for the possibility of an infinite number of values ​​of index k, in actual implementations there are only a finite number of values ​​of index k.

[0137] In the example MGA embodiment, using the new labels for the update vector elements σ[t,n] and the normalization amount f[t] defined above, the update vector elements σ[t+1,n] used to update the filter coefficients a[n] (determined for the previous time t) (at time t+1) are:

[0138]

[0139] where the symbol “·” denotes multiplication, σ[t,n] is the update vector element being updated (the update element used at the previous time t), γ is the smoothing factor (i.e., no smoothing in the case of γ=0), μ is the rate factor, and (f[t]) -1 / 2 is the normalization factor. Suitable values ​​for the rate factor μ and the smoothing factor γ are 0.005 and 0.6 respectively, assuming that the adaptation occurs 50 times per second for moderate digital signal levels for the microphone and the reference.

[0140] In general, the same rate factor μ may be used for each filter coefficient, or different values ​​of the rate factor μ may be used for each filter coefficient (μ in the equations in the previous paragraph may be written as "μ[n]" to explicitly indicate the rate factor for the "n"th filter coefficient). For example, each rate factor μ[n] may be one of the above-described weights μ[k] (where index k identifies the filter coefficient of the filter with tap index l in the above description of weights μ[k]). Alternatively or additionally, another weighting (e.g., a time-indexed weighting using the weights β[n] described below) may be applied to each update element σ[t+1,n] during adaptation, where such another weighting depends on which filter coefficients are being adjusted (e.g., so that different weights are applied to filter coefficients of different filters).

[0141] Using the update vector elements σ[t+1,n], the exemplary MGA embodiment updates the filter coefficients a[t,n] (determined for the previous time t) (at each time t+1) by utilizing smoothing of partial derivatives (as indicated in the above equation for σ[t+1,n]) and preferably utilizing time-indexed weighting. Specifically, the filter coefficient adaptation steps of the MGA exemplary embodiment are:

[0142] a[t+1,n]=a[t,n]–β[n]σ[t+1,n]

[0143] Where a[t+1,n] represents the updated prediction filter coefficients of the "n"th filter, and where β[n] is a time-index-based weight. Optionally, the time-index-based weighting is omitted (ie, each β[n] may have a value of 1).

[0144] Thus, during adaptation (at time t+1) of the current values ​​of the filter coefficients a[t,n] (determined at the previous time t), the adaptive MGA embodiment proceeds faster with larger absolute values ​​of β[n]σ[t+1,n] and slower with smaller absolute values ​​of β[n]σ[t+1,n].

[0145] With reference to the weights β[n], "time-index-based" weighting means that each weight β[n] depends on which filter coefficient (the "nth" filter coefficient) is being updated, where each index n corresponds to time. For example, each weight β[n] can be one of the above-mentioned weights μ[k], where index k corresponds to index n, because in the above description of weights μ[k], index k identifies the filter coefficient of the filter with tap index l (which in turn corresponds to time), so that weights μ[k] are based on the time index in the sense of distinguishing different filters of the filter bank.

[0146] In the adaptive MGA embodiment, it is clear that the update element σ[t+1,n] is obtained by normalizing and scaling each gradient assuming that the gradient has moved forward by some amount from its previous value And the adaptation is smoothed according to the smoothing factor γ. Each gradient By multiplying it by the normalization factor (f[t]) -1 / 2 ) and increases the adaptation step size when the prediction error decreases over time as expected, and decreases the adaptation step size when the prediction error does not decrease over time in the expected manner (e.g., in the case of unexpected or unpredicted noise). During each adaptation step, each gradient is scaled by the rate factor μβ[n] and normalized. As long as the movement of the scaled, normalized gradient still has a similar direction as the movement of the adjustment vector σ, the system will continue to increase the adaptation rate. If the gradient (or the scaled, normalized gradient) starts to behave unpredictably, for example as noise (e.g., due to the prediction filter coefficient a[n], for all or some values ​​of index n, approaching a minimum, and / or due to noise in the audio path), the adaptation rate will be reduced due to the low-pass (smoothing) nature of the update step. In other words, the MGA method speeds up the adaptive movement (adaptation rate) until the gradient All or some of the scaled, normalized versions of the gradients (which are considered together as a gradient vector for all values ​​of index n) start to become more random. Thus, the adaptation of the prediction filter coefficients is controlled based on the predictability of the adaptation direction and the adaptation gradients.

[0147] Figure 4 4 is a flow chart of an example process 400 for echo cancellation according to an embodiment of the present invention (e.g., smoothing using normalized gradient vectors, as in the adaptive MGA embodiment described above). Process 400 may be performed by an echo canceller system, which may include one or more appropriately programmed processors. The echo canceller may be implemented in (or as) a device (e.g., a mobile device) that includes a microphone and a speaker, and thus the echo canceller is sometimes referred to herein as a device.

[0148] refer to Figure 4, an echo canceller receives (410) an input signal from a microphone of a device. The echo canceller receives (420) an output signal (speaker feed) to a speaker on the same device as the microphone. The echo canceller predicts (430) a portion of the input signal (i.e., content) caused by audio content of the speaker feed. The prediction includes configuring an adaptive filter based on the input signal and the output signal. The configuration includes scaling (i.e., controlling) an adaptation rate of the adaptive filter based on at least predictability of an adaptation direction and an adaptation gradient. The echo canceller removes (440) from the input signal the portion of the input signal caused by the audio content of the speaker feed.

[0149] Example system architecture

[0150] Figure 5 is used to implement reference according to an embodiment Figure 1-4 A mobile device architecture (800) of features and processes described herein. According to an embodiment of the present invention, a device having architecture 800 may be configured (e.g., one or more processors 801 and audio subsystem 803 of the architecture may be configured) to perform echo cancellation (or steps thereof) by controlling a predictive filter adaptive step size. Architecture 800 may be implemented in any electronic device, including, but not limited to: desktop computers, consumer audio / video (AV) equipment, radio broadcasting equipment, mobile devices (e.g., smartphones, tablet computers, laptop computers, wearable devices). Figure 5 In the illustrated example embodiment, the architecture 800 is for a smartphone and includes one or more processors 801, peripheral device interfaces 802, audio subsystems 803, speakers 804, microphones 805, sensors 806 (e.g., accelerometers, gyroscopes, barometers, magnetometers, cameras), location processors 807 (e.g., GNSS receivers), wireless communication subsystems 808 (e.g., Wi-Fi, Bluetooth, cellular), and one or more I / O subsystems 809 (including touch controllers 810 and other input controllers 811), touch surfaces 812, and other input / control devices 813, which are coupled as shown. Other architectures with more or fewer components may also be used to implement the disclosed embodiments.

[0151] The memory interface 814 is coupled to the processor 801, the peripheral device interface 802, and the memory 815 (e.g., flash memory, RAM, ROM). The memory 815 stores computer program instructions and data, including but not limited to: operating system instructions 816, communication instructions 817, GUI instructions 818, sensor processing instructions 819, phone instructions 820, electronic message instructions 821, web browsing instructions 822, audio processing instructions 823, GNSS / navigation instructions 824, and application / data 825. The audio processing instructions 823 include instructions for executing reference Figure 1-4 Instructions that describe audio processing (including echo cancellation).

[0152] Aspects of the systems described herein can be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. The various parts of the adaptive audio system can include one or more networks, which include any desired number of individual machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0153] One or more of the components, blocks, processes, or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media, depending on their behavior, register transfers, logic components, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0154] Aspects of some embodiments of the present invention may be understood from one or more of the following example embodiments ("EEE"):

[0155] EEE 1. An echo cancellation method, comprising:

[0156] An echo canceller receives an input signal from a microphone;

[0157] An output signal of a loudspeaker is received by the echo canceller;

[0158] predicting, by the echo canceller, echo content of the input signal caused by acoustic emissions by the loudspeaker in response to the output signal, wherein the prediction comprises adapting at least one prediction filter with an adaptation step size controlled by using a gradient descent on a set of filter coefficients of the filter, wherein control of the adaptation step size is based at least in part on predictability of an adaptation direction and an adaptation gradient; and

[0159] At least some of the echo content that has been predicted during the prediction step is removed from the input signal.

[0160] EEE 2. A method according to EEE 1, wherein each adaptation step of the adaptation is based on a previously determined set of filter coefficients θ n-1Determine an updated set of filter coefficients θ n , which consists of subtracting an update term σ from a previously determined set of filter coefficients n , where the update term is at least partially determined by the adaptive gradient.

[0161] EEE 3. A method according to EEE 1 or EEE 2, wherein the adaptively determining at least one coefficient a[k] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,k] of the coefficient a[k] in response to a previously determined version a[t,k] of the coefficient a[k], wherein t represents time, and wherein:

[0162]

[0163] where X[t] is the time-varying weight, 1 / N is the normalization factor, |e[t]| is the absolute value of the prediction error e[t] at time t, is the adaptive gradient.

[0164] EEE 4. A method according to EEE 3, wherein the weight X[t] increases the adaptation step size when the prediction error decreases in an expected manner, and when the prediction error does not decrease in an expected manner, the adaptation speed is reduced, and wherein the normalization factor 1 / N is a dynamic normalization factor whose value increases when the adaptation approaches convergence.

[0165] EEE 5. A method according to EEE 3 or EEE 4, wherein X[t] = μ[k]s[t], wherein s[t] is a time-varying weight, μ[k] is a time-indexed weight of coefficient a[k], the prediction filter has a filter tap index l, and the weight μ[k] depends on the value of the filter tap index l.

[0166] EEE 6. The method according to EEE 1 or EEE 2, wherein the gradient descent is Nesterov accelerated gradient descent.

[0167] EEE 7. The method according to EEE 6, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein:

[0168] a[t+1,n]=a[t,n]–σ[t+1,n]

[0169] in

[0170]

[0171] where γ is the smoothing factor, μ is the factor, 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

[0172] EEE 8. The method according to EEE 6, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein:

[0173] a[t+1,n]=a[t,n]–β[n]σ[t+1,n]

[0174] Among them, β[n] is the weight based on the time index, where:

[0175]

[0176] Where γ is the smoothing factor, μ is the factor, and 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

[0177] EEE 9. A method according to EEE 8, wherein β[n] is a time-index-based weight of the coefficient a[n], the prediction filter including the coefficient a[n] has a filter tap index l, and the weight β[n] depends on the value of the filter tap index l. ...

[0178] EEE 10. The method according to any one of EEE 1 to EEE 9, wherein during adaptation of the prediction filter, the control of the adaptation step size is at least partially based on a filter tap index of the prediction filter. EEE 11.

[0179] EEE 11. A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of EEE 1 to EEE10.

[0180] EEE 12. A system configured to perform echo cancellation, the system comprising:

[0181] at least one processor coupled and configured to receive an input signal from a microphone and an output signal to a speaker and to determine at least one predictive filter in response to the input signal and the output signal,

[0182] wherein the at least one processor is configured to predict the echo content of the input signal caused by sound emissions of the loudspeaker in response to the output signal, comprising adapting at least one prediction filter with an adaptation step size, the adaptation step size being controlled by using a gradient descent on a set of filter coefficients of the filter, wherein the control of the adaptation step size is based at least in part on the predictability of an adaptation direction and an adaptation gradient, and

[0183] The at least one processor is coupled and configured to process the input signal to remove at least some of the echo content that has been predicted from the input signal.

[0184] EEE 13. A system according to EEE 12, wherein each adaptation step of the adaptation starts from a previously determined set of filter coefficients θ n-1 Determine an updated set of filter coefficients θ n , which consists of subtracting an update term σ from a previously determined set of filter coefficients n , where the update term is at least partially determined by the adaptive gradient.

[0185] EEE 14. A system according to EEE 12 or EEE 13, wherein the adaptation determines at least one coefficient a[k] of the filter, each adaptation step of the adaptation determining an updated version a[t+1,k] of the coefficient a[k] in response to a previously determined version a[t,k] of the coefficient a[k], wherein t represents time, and wherein:

[0186]

[0187] where X[t] is the time-varying weight, 1 / N is the normalization factor, |e[t]| is the absolute value of the prediction error e[t] at time t, is the adaptive gradient.

[0188] EEE 15. A system according to EEE 14, wherein the weight X[t] increases the adaptation step size when the prediction error decreases in an expected manner, and when the prediction error does not decrease in an expected manner, the adaptation speed is reduced, and wherein the normalization factor 1 / N is a dynamic normalization factor whose value increases when the adaptation approaches convergence.

[0189] EEE 16. A system according to EEE 14 or EEE 15, wherein X[t] = μ[k]s[t], where s[t] is a time-varying weight, μ[k] is a time-indexed weight of coefficient a[k], the prediction filter has a filter tap index l, and the weight μ[k] depends on the value of the filter tap index l.

[0190] EEE 17. A system according to EEE 12 or EEE 13, wherein the gradient descent is Nesterov accelerated gradient descent.

[0191] EEE 18. A system according to EEE 17, wherein the adapting determines at least one coefficient a[n] of the filter, each adaptation step of the adapting determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein:

[0192] a[t+1,n]=a[t,n]–σ[t+1,n]

[0193] in

[0194]

[0195] where γ is the smoothing factor, μ is the factor, 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

[0196] EEE 19. The system according to EEE 17, wherein the adapting determines at least one coefficient a[n] of the filter, each adaptive step of the adapting determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein:

[0197] a[t+1,n]=a[t,n]–β[n]σ[t+1,n]

[0198] Among them, β[n] is the weight based on the time index, where:

[0199]

[0200] Where γ is the smoothing factor, μ is the factor, and 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

[0201] EEE 20. A system according to EEE 19, wherein β[n] is a time-index-based weight of the coefficient a[n], the prediction filter including the coefficient a[n] has a filter tap index l, and the weight β[n] depends on the value of the filter tap index l. ...

[0202] EEE 21. A system according to any one of EEE 12 to EEE 20, wherein during adaptation of the prediction filter, control of the adaptation step size is at least partially based on a filter tap index of the prediction filter. EEE 22.

[0203] Although one or more implementations have been described by way of example and according to specific embodiments, it should be understood that one or more implementations are not limited to the disclosed embodiments. On the contrary, the present invention is intended to cover various modifications and similar arrangements that are obvious to those skilled in the art. Therefore, the scope of the appended claims should be consistent with the broadest interpretation so as to cover all such modifications and similar arrangements.

Claims

1. An echo cancellation method, comprising: An echo canceller receives an input signal from a microphone; An output signal of a loudspeaker is received by the echo canceller; predicting, by the echo canceller, echo content of the input signal caused by acoustic emissions by the loudspeaker in response to the output signal, wherein the prediction comprises adapting at least one prediction filter with an adaptation step size controlled by using a gradient descent on a set of filter coefficients of the filter, wherein control of the adaptation step size is based at least in part on predictability of an adaptation direction and an adaptation gradient; and removing from the input signal at least some of the echo content that has been predicted during the prediction step, wherein the adapting determines at least one coefficient a[k] of the filter, each adaptation step of the adapting determining an updated version a[t+1,k] of the coefficient a[k] in response to a previously determined version a[t,k] of the coefficient a[k], wherein t represents time, and wherein: where X[t] is the time-varying weight, 1 / N is the normalization factor, │e[t]│ is the absolute value of the prediction error e[t] at time t, is the adaptive gradient.

2. The method according to claim 1, wherein each adaptation step of the adaptation starts from a set of filter coefficients θ previously determined. n-1 Determine an updated set of filter coefficients θ n , which consists of subtracting an update term σ from a previously determined set of filter coefficients n , where the update term is at least partially determined by the adaptive gradient.

3. The method according to claim 1, wherein: When the prediction error decreases in an expected manner, the weight X[t] increases the adaptive step size, and when the prediction error does not decrease in an expected manner, the adaptive speed is reduced, and wherein the normalization factor 1 / N is a dynamic normalization factor whose value increases when the adaptation approaches convergence.

4. A method according to claim 1, wherein X[t] = μ[k]s[t], where s[t] is a time-varying weight, μ[k] is a time-indexed weight of coefficient a[k], the prediction filter has a filter tap index l, and the weight μ[k] depends on the value of the filter tap index l.

5. The method of claim 1, wherein the gradient descent is Nesterov accelerated gradient descent.

6. The method according to claim 5, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein: a[t+1,n]=a[t,n]–σ[t+1,n] in where γ is the smoothing factor, μ is the factor, 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

7. The method of claim 5, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein: a[t+1,n]=a[t,n]–β[n]σ[t+1,n] in, β[n] is a weight based on the time index, where: Where γ is the smoothing factor, μ is the factor, and 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

8. A method according to claim 7, wherein β[n] is a time-indexed weight of the coefficient a[n], the prediction filter including the coefficient a[n] has a filter tap index l, and the weight β[n] depends on the value of the filter tap index l.

9. The method according to claim 1, Wherein during adaptation of the prediction filter, control of the adaptation step size is based at least in part on a filter tap index of the prediction filter.

10. A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 9.

11. A system configured to perform echo cancellation, the system comprising: at least one processor coupled and configured to receive an input signal from a microphone and an output signal to a speaker and to determine at least one prediction filter in response to the input signal and the output signal, in, The at least one processor is configured to predict an echo content of the input signal caused by a sound emission by the loudspeaker in response to the output signal, comprising adapting a prediction filter with an adaptation step size controlled by using a gradient descent on a set of filter coefficients of the filter, wherein control of the adaptation step size is based at least in part on a predictability of an adaptation direction and an adaptation gradient, and wherein the at least one processor is coupled and configured to process the input signal to remove at least some of the echo content that has been predicted from the input signal, wherein the adapting determines at least one coefficient a[k] of the filter, each adaptation step of the adapting determining an updated version a[t+1,k] of the coefficient a[k] in response to a previously determined version a[t,k] of the coefficient a[k], wherein t represents time, and wherein: where X[t] is the time-varying weight, 1 / N is the normalization factor, │e[t]│ is the absolute value of the prediction error e[t] at time t, is the adaptive gradient.

12. The system of claim 11, wherein each adaptation step of the adaptation starts from a previously determined set of filter coefficients θ n-1 Determine an updated set of filter coefficients θ n , which consists of subtracting an update term σ from a previously determined set of filter coefficients n , where the update term is at least partially determined by the adaptive gradient.

13. The system according to claim 11, wherein: When the prediction error decreases in an expected manner, the weight X[t] increases the adaptive step size, and when the prediction error does not decrease in an expected manner, the adaptive speed is reduced, and wherein the normalization factor 1 / N is a dynamic normalization factor whose value increases when the adaptation approaches convergence.

14. A system according to claim 11, wherein X[t] = μ[k]s[t], where s[t] is a time-varying weight, μ[k] is a time-indexed weight of coefficient a[k], the prediction filter has a filter tap index l, and the weight μ[k] depends on the value of the filter tap index l.

15. The system of claim 11, wherein the gradient descent is Nesterov accelerated gradient descent.

16. The system of claim 15, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein: a[t+1,n]=a[t,n]–σ[t+1,n] in where γ is the smoothing factor, μ is the factor, 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

17. The system of claim 15, wherein the adaptively determining at least one coefficient a[n] of the filter, each adaptive step of the adaptively determining an updated version a[t+1,n] of the coefficient a[n] in response to a previously determined version a[t,n] of the coefficient a[n], wherein t represents time, and wherein: a[t+1,n]=a[t,n]–β[n]σ[t+1,n] in, β[n] is a weight based on the time index, where: Where γ is the smoothing factor, μ is the factor, and 1 / (f[t]) 1 / 2 is the normalization factor, e 2 [t] is the squared prediction error at time t, is the adaptive gradient.

18. A system according to claim 17, wherein β[n] is a time-indexed weight of the coefficient a[n], the prediction filter including the coefficient a[n] has a filter tap index l, and the weight β[n] depends on the value of the filter tap index l.

19. The system according to claim 11, Wherein during adaptation of the prediction filter, control of the adaptation step size is based at least in part on a filter tap index of the prediction filter.

20. An echo cancellation device, comprising: at least one processor; as well as A non-transitory computer readable medium storing instructions, which when executed by at least one processor causes the at least one processor to perform the method according to any one of claims 1-9.

21. A computer program product comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1-9.

22. An echo cancellation device comprising means for performing the method according to any one of claims 1-9.

Images