Speech enhancement and interference suppression

Abstract

Methods, systems, and media for processing audio are provided. In some embodiments, a method involves receiving an input audio signal from a plurality of microphones. The method may involve identifying an angle of arrival associated with the input audio signal. The method may involve determining a plurality of gains corresponding to a plurality of frequency bands of the input audio signal based on a combination of at least: 1) a representation of a covariance of a signal associated with a microphone of the plurality of microphones on a per frequency band basis; and 2) an angle of arrival. The method may involve applying the plurality of gains to a plurality of frequency bands of the input audio signal such that at least a portion of the input audio signal is suppressed to form an enhanced audio signal.

Description

Technical Field

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 355,328 filed on June 24, 2022 and U.S. Provisional Patent Application No. 63 / 489,347 filed on March 9, 2023, each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to systems, methods, and media for speech enhancement and interference suppression. Background Art

[0003] In various audio applications, such as audio conferencing techniques, it may be difficult to hear the audio of a speaker of interest, particularly when the speaker of interest is competing with other noise and / or speakers. For example, in various audio conferencing techniques, multiple microphones may be used to pick up audio from multiple speakers in a room, and it may be difficult to isolate and / or emphasize the speaker of interest relative to noise associated with other areas and / or microphones.

[0004] Symbols and terminology

[0005] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers). A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

[0006] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0007] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0008] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the invention

[0009] Methods, systems, and media for speech enhancement and interference suppression are provided. In some embodiments, a method may involve receiving an input audio signal from a plurality of microphones. The method may involve identifying an angle of arrival associated with the input audio signal. The method may involve determining a plurality of gains corresponding to a plurality of frequency bands of the input audio signal based on a combination of at least the following: 1) a representation of the covariance of a signal associated with a microphone in the plurality of microphones on a per frequency band basis; and 2) an angle of arrival. The method may involve applying the plurality of gains to the plurality of frequency bands of the input audio signal such that at least a portion of the input audio signal is suppressed to form an enhanced audio signal.

[0010] In some examples, identifying the angle of arrival includes converting a received signal associated with a microphone of the plurality of microphones into a spatial representation, and wherein the input audio signal corresponds to the spatial representation.

[0011] In some examples, determining the plurality of gains comprises: identifying one or more objects of the input audio signal; and clustering the one or more objects of the input audio signal into one of a plurality of clusters, wherein the plurality of gains associated with the current time frame of the input audio signal are determined based on proximity of the current time frame of the input audio signal to objects within the cluster of the one or more objects. In some examples, identifying the one or more objects of the input audio signal is based on current input and historical input. In some examples, clustering the one or more objects of the input audio signal is responsive to determining that the number of frames in which the one or more audio objects appear in the input audio signal has exceeded a threshold number of frames. In some examples, clustering a given object of the one or more objects of the input audio signal comprises one of the following: 1) updating an existing object in a cluster; 2) creating a new object corresponding to the given object in the cluster; or 3) replacing an existing object in the cluster with the given object. In some examples, the replaced existing object is an existing object with the lowest activity level in the cluster. In some examples, the clustering is performed on a broadband basis for the multiple frequency bands. In some examples, clustering the one or more objects includes determining a plurality of similarity indicators of the input audio signal to each cluster. In some examples, the plurality of similarity indicators correspond to the plurality of frequency bands. In some examples, determining the similarity indicator of a given cluster is based on the most active object within the given cluster. In some examples, the plurality of gains are determined using the plurality of similarity indicators. In some examples, the plurality of clusters include clusters within a region of interest and clusters outside the region of interest. In some examples, a method further involves determining, for each of the plurality of frequency bands, a lower gain limit applicable to a portion of the input audio signal within the region of interest and an upper gain limit applicable to a portion of the input audio outside the region of interest, wherein the plurality of gains are limited by the lower gain limit and the upper gain limit.

[0012] In some examples, applying the multiple gains includes: filtering the input audio signal using a linear filter to generate a filtered signal; grouping the input audio signal and the filtered signal into the multiple frequency bands; calculating multiple gains for the multiple frequency bands by taking the difference between the power of the input audio signal and the filtered signal; determining multiple gain limits; clamping the gains to the gain limits; and applying the clamped gains to the input audio signal.

[0013] In some examples, applying the plurality of gains includes: determining a ratio of spatial components of the input audio signal; and applying the plurality of gains based at least in part on the ratio of the spatial components.

[0014] In some examples, a method further involves smoothing the plurality of gains prior to applying the plurality of gains. In some examples, a method further involves rendering the enhanced audio signal via a loudspeaker or headphones.

[0015] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0016] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0017] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a diagram illustrating a region of interest according to some embodiments.

[0019] Figure 2 is a schematic block diagram of a system for enhancing speech within a region of interest and suppressing interference outside the region of interest, according to some embodiments.

[0020] Figure 3 is a flow chart of an example process for enhancing speech within a region of interest and suppressing interference outside the region of interest in accordance with some embodiments.

[0021] Figure 4 is a flow chart of an example process for clustering audio objects in accordance with some embodiments.

[0022] Figure 5A is a flow chart of an example process for determining and utilizing a similarity index for clustering audio objects, according to some embodiments.

[0023] Figure 5B is a flow chart of an example process for determining a similarity index for audio objects according to some embodiments.

[0024] Figure 5C is a flow diagram of an example process for updating an object ranking according to some embodiments.

[0025] Fig. 6A and Figure 6B is a schematic block diagram of an example beamforming system in accordance with some embodiments.

[0026] Figure 7 is a flow chart of an example process for enhancing speech in accordance with some embodiments.

[0027] Figure 8 A block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0028] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0029] It may be difficult to accurately enhance audio signals of interest and suppress audio signals of no interest (e.g., interfering audio signals). For example, in an audio conferencing system or a video conferencing system, it may be desirable to enhance speech associated with a given speaker while suppressing speech, noise, etc. from other speakers. Conventional noise suppression techniques may utilize beamforming techniques to suppress audio outside the region of interest. However, conventional techniques may over-suppress the audio of interest and under-suppress interfering audio of no interest, particularly when the audio to be suppressed includes competing speakers or other speech-like signals.

[0030] Disclosed herein are systems, methods, and media for speech enhancement. Using the techniques disclosed herein, gain can be determined and applied on a per-band basis. In some embodiments, gain can be determined based on the angle of arrival of the input audio signal and the covariance between the signals of different microphones on all frequency bands of the system. Covariance is generally referred to as a power vector in this article. In some embodiments, audio objects (generally referred to as "objects" herein) in the input audio signal can be clustered based on the angle of arrival and the power vector. For example, objects can be clustered into clusters within a region of interest or clusters outside the region of interest. Gain can be determined on a per-band basis and using clustering. By determining gain using clustering of audio objects, the techniques disclosed herein can avoid over-suppression of signals associated with objects within the region of interest, and avoid insufficient suppression of signals associated with objects outside the region of interest. By determining and applying gain on a per-band basis, gain can be robustly applied even when the signal to be suppressed includes competing voices (e.g., competing with the voice to be enhanced).

[0031] Figure 1 1 is a diagram illustrating a region of interest according to some embodiments. System 100 may include one or more microphones. For example, system 100 may be part of a video conferencing system or an audio conferencing system. System 100 may be associated with region of interest 102. As illustrated, in some embodiments, region of interest 102 may be a sector originating from system 100. The area surrounding region of interest 102 corresponds to external region 104. Using the techniques described herein, system 100 may enhance speech originating from speaker 106 within region of interest 102 while suppressing speech or noise originating from external region 104, such as speech from speaker 108.

[0032] In some embodiments, the technology described herein may apply beamforming techniques to signals from one or more microphones to suppress signals outside the region of interest and enhance signals within the region of interest. Beamforming techniques may be applied by determining a gain on a per-band basis. Applying a gain on a per-band basis rather than on a broadband basis may allow for more accurate suppression of signals outside the region of interest in a scene with competing speakers (e.g., competing voice signals) (such as in the context of an audio conference or video conference). Gains may be determined using acoustic or audio scene analysis techniques. Specifically, objects within an audio signal may be clustered as belonging to one of a plurality of clusters. In one example, a plurality of clusters may include clusters within the region of interest and clusters outside the region of interest. In another example, a plurality of clusters may include clusters within the region of interest, clusters outside the region of interest, and transition zone clusters. Gains may then be determined for each cluster on a per-band basis. In some embodiments, scene analysis may be performed by estimating the angle of arrival of an incoming audio signal. In some embodiments, scene analysis may be performed on a sub-band version of the incoming audio signal to achieve gain determination on a per-band basis. In some embodiments, gains may be determined based on a power vector, which may indicate the covariance of signals associated with different microphones on a per-band basis. Note that because scene analysis and gain determination can be performed on a per-band basis, the speech of competing speakers can be effectively enhanced or suppressed depending on the direction of interest, allowing for more effective and robust noise suppression even in the presence of multiple speakers or competing speech signals.

[0033] Figure 2 is a schematic diagram of an example system 200 for applying beamforming techniques to an input audio signal according to some implementations. The blocks of system 200 may be implemented on a user device such as a laptop computer, a desktop computer, a computer or processor associated with an audio or video conferencing system, etc. Example components of such a device are shown in FIG. Figure 8 FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure.

[0034] As illustrated, the system 200 may acquire audio signals from a set of microphones (e.g., from one microphone, from two microphones, from five microphones, etc.). The set of input audio signals is generally referred to herein as M1(t), M2(t), ..., M of the N microphones. N (t). The input audio signal from the microphone may first be processed by a short-time Fourier transform (STFT) block 202. The STFT block 202 may transform the input audio signal from the time domain to the frequency domain. For example, for a given frame n, the frequency domain representation generated by the STFT block 202 may be represented as: M1(n,k), M2(n,k), ..., MN (n, k), where k = 0, 1, ... K-1, and represents the discrete Fourier transform (DFT) bin index of the total K bins. The frequency domain signal generated by the STFT block 202 can then be passed to the high-fidelity stereo conversion block 204, the power vector block 206, and the beamforming block 214, as shown in FIG. Figure 2 shown.

[0035] The high-fidelity stereo conversion block 204 can convert the frequency domain audio signal into a scene-based audio signal. Specifically, the high-fidelity stereo conversion block 204 can transform the frequency domain audio signal into a high-fidelity stereo format including, for example, an X component corresponding to the front-to-back direction, a Y component corresponding to the left-to-right direction, and a W component corresponding to the omnidirectional component. Note that although a first-order high-fidelity stereo format is generally used herein, other spatial coding formats may also be used. For a given frame n and a given DFT bin index k, the high-fidelity stereo conversion block 204 can generate a high-fidelity stereo representation of the input audio signal represented by {W(n,k), X(n,k), Y(n,k)}.

[0036] The power vector block 206 may be configured to determine the covariance between microphone signal pairs on a per-band basis. For a given frame n, the power vector may be represented as v(n). More detailed techniques for determining the power vector are described below in conjunction with Figure 3 Give a description.

[0037] The high-fidelity stereo representation of the audio signal for a given frame n may be passed to the angle estimation block 208 and the band splitting block 210. The angle estimation block 208 may be configured to determine the angle of arrival of the frame n of the input audio signal. The angle of arrival is generally denoted herein as θ. More detailed techniques for determining the angle of arrival are described below in conjunction with Figure 3 Give a description.

[0038] The band splitting block 210 can be configured to separate the frequency domain representation of the frame n of the input audio signal into a plurality of frequency bands. In some embodiments, the band splitting can be in a non-uniform bandwidth frequency band domain, for example, to simulate the frequency processing of the human cochlea. The results of the band splitting block 210 can be utilized by the beamforming block 214, as will be described below.

[0039] The audio scene analysis block 212 may receive the angle of arrival θ and the power vector v. Based on the angle of arrival θ and the power vector v, the acoustic scene analysis block 212 may determine a gain limit on a per-band basis. The gain limit may be determined with respect to a plurality of regions (e.g., an in-region region and an out-of-region region), such as in Figure 1 A more detailed technique for determining gain bounds for a set of regions is described in Figure 35 and described below in conjunction with these figures. Note that, as used herein, gain values ​​are generally negative values. Additionally, note that gain bounds can include, for example, a lower gain limit applicable to audio objects within a region of interest, where the lower gain limit specifies a maximum gain to be applied to objects within the region of interest, thereby preventing over-suppression of objects within the region of interest. Additionally or alternatively, gain bounds can include an upper gain limit applicable to audio objects outside the region of interest, where the upper gain limit specifies a minimum gain to be applied to objects outside the region of interest, thereby preventing under-suppression of objects outside the region of interest.

[0040] The gain limits determined by the audio scene analysis block 212 may be passed to the beamforming block 214. Using the determined gain limits and the sub-band information determined by the sub-band block 210, the beamforming block 214 may be configured to apply the gain limits to a given frame n of the input audio signal. Note that the gain limits may be applied on a per-band basis rather than on a wideband basis. In some embodiments, the beamforming block 214 may perform a smoothing process on the gains before applying the gains. An example system that may be implemented as the beamforming block 214 is Fig. 6A and Figure 6B The diagrams are shown in FIG. 1 and described below in conjunction with these figures.

[0041] The modified audio signal (e.g., with the gain applied) can then be passed to the inverse STFT block 216. The inverse STFT block 216 can transform the modified audio signal in the frequency domain into an enhanced audio signal in the time domain. For N microphones, the enhanced audio signal can be represented herein as: m'1(t), m'2(t), ..., m' N (t).

[0042] As described above, beamforming techniques may be applied to perform accurate and robust noise suppression and enhancement of speech of interest, particularly in situations with multiple talkers or competing talkers. Gains may be determined for each of a set of regions or clusters on a per-band basis. For example, the clusters may correspond to clusters within the region of interest and clusters outside the region of interest, such as in Figure 1 , and described above in conjunction with the figure. The gain may be determined based on a gain limit. The gain limit, in turn, may be determined based on a similarity index that indicates the similarity of a current frame of the input audio signal to a set of clusters on a per frequency band basis. Note that audio objects may be created and clustered based on historical angles of arrival of frames of the input audio signal and a power vector representing the covariance between two signals from two microphones on a per frequency band basis. By applying a gain on a per frequency band basis, competing speech outside the region of interest may be more effectively suppressed.

[0043] Figure 3 300 is a flow chart of an example process 300 for speech enhancement according to some implementations. The blocks of process 300 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a computer or processor associated with an audio or video conferencing system, etc.). Example components of such a device are described in Figure 8 An example of a processor or controller that may be used is a control system 810, such as in Figure 8 In some embodiments, the blocks of process 300 may be arranged in a manner different from that of Figure 3 In some implementations, two or more blocks of process 300 may be performed substantially in parallel. In some implementations, one or more blocks of process 300 may be omitted.

[0044] Process 300 may begin at 302 by determining the spatial components of an input audio signal. Figure 2 As described above, the input audio signal may be associated with a set of N microphones. The set of input audio signals may be represented as m1(t), m2(t), ..., m N (t). Before determining the spatial components, the input audio signal can be transformed into the frequency domain, for example using STFT. The frequency domain signal can be represented as: M1(n,k), M2(n,k), ..., M N (n, k), where k = 0, 1, ... K-1, and represents the discrete Fourier transform (DFT) bin index for a total of K bins. The spatial components of the input audio signal can then be determined by performing an Ambisonics conversion (e.g., from a frequency domain representation). For example, for a given frame n, the Ambisonics representation can be given by: {W(n, k), X(n, k), Y(n, k)}, where W represents the omnidirectional component, X represents the front-to-back component, and Y represents the left-right component.

[0045] At 304, process 300 may estimate the angle of arrival of the input audio signal, generally denoted herein as θ. For example, the angle of arrival may be estimated based on spatial components. As a more specific example, given a high-fidelity stereo representation including W, X, and Y components, the angle of arrival may be estimated based on a covariance matrix of the components. For example, the angle of arrival may be estimated by performing a principal component analysis (PCA) on the covariance matrix. It will be appreciated that, given the W, X, and Y sound components, the sound field represented by these components may be represented as:

[0046]

[0047] In the equation given above, S represents the sound source component, and i is an index representing the component number. In the example given above, at most three components can be separated from (W, X, Y), so i can iterate over 0, 1, and 2.

[0048] Therefore, the covariance matrix for a given frame n can be expressed as:

[0049]

[0050] In the equation given above, α is a smoothing parameter, k represents the bin index, and W*, X*, and Y* represent the complex conjugates of W, X, and Y, respectively.

[0051] The covariance matrix Cov0(n) can have an eigenvector with the largest eigenvalue Then, the angle of arrival can be estimated based on the maximum eigenvalue of the covariance matrix. In one example, the angle of arrival θ can be determined by:

[0052]

[0053] At 306, process 300 may determine a power vector associated with the input audio signal. The power vector may indicate the covariance between signals associated with different microphones on a per-band basis. Given a rectangular frequency band, where each bin k contributes exactly one output frequency band with a gain of 1, the covariance (generally denoted herein as Cov n (b)) can be determined by the following formula:

[0054]

[0055] In the equation given above, Note that B(b) represents a set of all STFT bins (eg, subbands) belonging to frequency band b, and α is a weighting factor configured to consider the weighted contributions of past covariance and estimated covariance. H (n, k) is a vector of length N, where N is the number of microphones and K is the number of bins.

[0056] In some embodiments, M(n, k) can be expressed as [M1(n, k), M2(n, k), ... M N (n,k)]^T.

[0057] Note that for a bin k, the weights w on all bands b k,b The sum must be 1. Therefore, the covariance of non-rectangular bands can be calculated using weights w for different bands and bins. k,b Note that linear, logarithmic, or mel-frequency cosine bands, triangle bands can be used. The covariance Cn (b) You can use the above method for Cov n (b) is determined by the equation given, but using where Σ k＝B(b) w k，b =1.

[0058] As used herein, the power vector for a given frequency band b is generally denoted as v b , where v b is the normalized covariance matrix. In some embodiments, the normalized covariance matrix may be the covariance matrix divided by its trace, so that the normalized covariance matrix represents only the directions with all horizontal components removed. For example, for a system with three microphones, v b It can be determined by the following formula:

[0059]

[0060] In some embodiments, the first element of each frequency band may be 1.0 and thus may be removed. The length of the power vector may be (N 2 -1)*B, where B is the total number of frequency bands. The power vector v b may be the power vector of band b with the first element removed, so it may have N 2 -1 real-valued element.

[0061] In 308, the process 300 may create and cluster objects of the input audio signal on a per-band basis for a set of frequency bands based on the angle of arrival and the power vector. In some embodiments, clustering may be performed based on the angle of arrival (e.g., as determined in block 304) and the power vector (e.g., as determined in block 306). In some embodiments, the group of clusters may include two clusters—a cluster within the region of interest and a cluster outside the region of interest. In some embodiments, the group of clusters may include three clusters, such as a cluster within the region of interest, a cluster outside the region of interest, and a transition zone cluster, the transition zone cluster corresponding to a zone or region between the cluster within the region of interest and the cluster outside the region of interest. Note that, although two clusters are generally described herein, the group of clusters may include any suitable number of clusters (e.g., two, three, five, ten, etc.). In addition, it should be noted that in some embodiments, there may be a maximum number of objects that are allowed to be assigned to a given cluster. Therefore, each cluster may include the M most prominent or important objects within the cluster, where M represents the maximum allowable number of objects allowed in a given cluster. Note that the M most prominent or important objects may be determined, identified, or selected based on the ranking of the objects. Example techniques for determining object rankings are Figure 5B and Figure 5C, and described below in conjunction with these figures. Example values ​​of objects that may be allowed in a given cluster include two, three, five, ten, etc. Figure 4 is a flow chart depicting an example process for clustering objects of an input audio signal according to some implementations.

[0062] At 310, process 300 may determine a similarity index between the current input audio signal and each cluster on a per frequency band basis. For a given cluster i and a given frequency band b, the similarity index may be represented as c i,b For example, given two clusters (e.g., a cluster inside the region of interest and a cluster outside the region of interest), for frequency band b, the similarity index can be expressed as c in,b and c out,b The similarity index for a given cluster and a given frequency band may be based on the power vectors of objects assigned to the cluster and the given frequency band. More detailed example techniques for determining the similarity index are described in Figure 5B FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure.

[0063] At 312, process 300 may determine a gain bound for each frequency band based on the similarity index of each cluster. The gain for a given frequency band b may be denoted herein as g b . In some embodiments, the gain of a given frequency band can be determined based on the gain upper and gain lower limits determined for each cluster and each frequency band. For example, in some embodiments, a gain lower limit can be set for each frequency band of a cluster within the region of interest, thereby ensuring that the gain is at least a predetermined level corresponding to the gain lower limit. Continuing with this example, a gain upper limit can be set for each frequency band of a cluster outside the region of interest, thereby ensuring that the gain of each frequency band is at least a certain minimum level corresponding to the gain upper limit. Note that as used herein, gain is a negative number. For example, given two clusters (e.g., a cluster within the region of interest and a cluster outside the region of interest), the gain lower limit of frequency band b of the cluster within the region can be determined by the following formula:

[0064]

[0065] Continuing with this example, the upper limit of the frequency band b of the cluster outside the region of interest can be determined by:

[0066]

[0067] In the equations given above, β0, β1, and β2 are all constants. In some embodiments, G L The value can be greater than G U In one example, G L can be -10dB, while G UIt can be -30dB. In another example, G L can be -20dB, while G U In some embodiments, the gain for a given frequency band can be based on an upper gain limit and a lower gain limit (e.g., as described above) and a first pass beamforming gain for the frequency band determined based on processing of the input audio signal by the beamforming system. Example beamforming systems and techniques for determining the gain for a given frequency band based on an upper gain limit and a lower gain limit are described in Fig. 6A and Figure 6B The diagrams are shown in FIG. 1 and described below in conjunction with these figures.

[0068] It should be understood that the clustering of objects (e.g., as described above in connection with block 308), the determination of similarity indicators for each cluster and each frequency band (e.g., as described above in connection with block 310), and the determination of upper and lower gain limits for each frequency band (e.g., as described above in connection with block 312) can be implemented by a scene analysis block, for example, as described in Figure 2 , and described above in conjunction with this figure.

[0069] At 314, the process 300 can apply gain limits to the banded gains on a per-band basis. The gain limits can be applied to a beamforming process. It should be noted that in some embodiments, the gains can be smoothed (e.g., with respect to time or frames of the audio signal) before being applied to prevent discontinuous jumps in the applied gains. The process 300 can generate a modified audio signal in the frequency domain by applying the gains on a per-band basis.

[0070] In 316, the process 300 can generate an output audio signal, sometimes referred to herein as an enhanced audio signal. For example, the process 300 can transform the modified audio signal (e.g., generated in block 314) into the time domain to generate the output audio signal. As a more specific example, in some embodiments, the process 300 can apply the inverse STFT to the modified audio signal to generate the output audio signal. Note that the output audio signal can be a signal in which at least a portion of the input audio signal is suppressed. The suppressed portion can correspond to speech or noise outside the region of interest. Therefore, the output audio signal can be considered to be an enhanced audio signal.

[0071] As above combined Figure 2 and Figure 3As described above, an object of an input audio signal can be clustered as belonging to one of a set of clusters. In some embodiments, when performing clustering, the historical context of the object of the input audio signal can be taken into account. For example, in some embodiments, an object can be assigned to a cluster only if the number of frames in which it appears in the input audio signal exceeds a predetermined number of frames or the duration in the input audio signal exceeds a predetermined duration. In some embodiments, temporary objects (generally denoted as . t ) and the current object (generally denoted as O in this article) c ) to take into account the historical context of the object, the temporary object tracking object in the input audio signal that has not yet appeared in the input audio signal for a minimum duration or a minimum number of frames, and the current object tracking object in the input audio signal that appears for more than the minimum duration or for more than the minimum number of frames but has not yet been assigned to a cluster.

[0072] Figure 4 is a flow chart of an example process 400 for clustering an object O of an input audio signal according to some embodiments. Note that the object O is associated with a feature space formed by the angle of arrival associated with the object and the power vector associated with the object. The feature space can be formed by In some embodiments, the blocks of process 400 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a computer or processor associated with an audio or video conferencing system, etc.). An example of such a processor or controller is control system 810, such as in Figure 8 In some embodiments, the blocks of process 400 may be arranged in a manner different from that of Figure 4 In some implementations, two or more blocks of process 400 may be performed substantially in parallel. In some implementations, one or more blocks of process 400 may be omitted.

[0073] Process 400 may begin at 402 by creating a temporary object O t and the current object O c Set to null or empty state respectively. Process 400 can then proceed to point A. Figure 4 As shown, process 400 may loop back to point A after various operations.

[0074] At 404, process 400 may determine c In other words, process 400 can determine whether temporary object O is null during the previous iteration of process 400. t Whether the previous version of O appears in the input audio signal for enough frames to be assigned as the current object Oc At 404, if process 400 determines that c is null (i.e., not previously assigned in a previous iteration of process 400). c ), or "yes" in 404, then process 400 can proceed to block 408 and determine whether t In other words, process 400 can determine whether the object has not been assigned to the current object ( c ) to determine whether the temporary object assigned to the cluster has been initialized. In 404, if process 400 determines that t is null ("yes" in 404), process 400 can proceed to 406 and can initialize and update O t For example, to initialize and update O t , process 400 may set O t , so that the feature space of the object in the current frame of the input audio signal has an arrival angle θ and a power vector v.

[0075] Conversely, at 408, if process 400 determines that t If is not null, process 400 may proceed to block 410 and may determine whether t Is the distance between the angle of arrival (denoted as θ) associated with the object in the input audio signal frame less than a minimum distance threshold (denoted as d min ). In some embodiments, the distance may be determined as:

[0076]

[0077] At 410, if process 400 determines that t If the distance between the angle of arrival of the object in the input audio signal frame and the minimum distance threshold is not less than the minimum distance threshold (“No” in 410), the process 400 may proceed to block 412 and may t In other words, process 400 can be responsive to determining that t The distance between the angle of arrival and the target object exceeds the minimum distance threshold and determines that the current object does not correspond to the previously stored target object. t Conversely, at 410, if process 400 determines that t If the distance between the angle of arrival of the object in the input audio signal frame is less than the minimum distance threshold (“yes” in 410), the process 400 may proceed to block 414 and may determine whether t In other words, the process 400 can determine whether the age of the object of the input audio signal frame (corresponding to O) is less than a first time threshold (generally denoted as T0 in this article). t) whether there has been a duration corresponding to the first time threshold T0. In 414, if the process 400 determines that t If the age of the user is less than the first time threshold T0 (“yes” in 414), the process 400 may proceed to block 418 and may update the value of O. t For example, update O t May involve updating O t To combine the current arrival angle θ and power vector v associated with the current frame of the input audio signal and t The associated θ and v. In one example, the object O can be updated by:

[0078]

[0079] Conversely, at 414, if process 400 determines that t If the age of O exceeds the first time threshold T0 (“No” in 414), the process 400 may proceed to block 416 and may c Set to 0 t , and O t Set to null or empty. In other words, once process 400 determines that temporary object O t If the temporary object has been present in the input audio signal for more than the first time threshold, the process 400 can promote the temporary object to the current object to be clustered and reset the temporary object to null or empty in order to track the new temporary object. The process 400 can then proceed back to point A.

[0080] Referring back to block 404, at 404, if process 400 determines that c If the value is not null or empty (“No” in 404), process 400 may proceed to block 420 and may determine whether c Is the distance between the arrival angle of the object of the current frame of the input audio signal less than the minimum distance threshold d min . For example, the distance can be determined by:

[0081]

[0082] At 420, if process 400 determines that c If the distance between the angle of arrival of the object of the current frame and the minimum distance threshold is not less than the minimum distance threshold (“No” in 420), the process 400 may proceed to block 422 and may c and O t are set to null or empty. Process 400 can then return to point A so that another temporary object can be tracked and, optionally, eventually promoted to the current object. Conversely, at 420, if process 400 determines thatc If the distance between the angle of arrival of the object in the current frame and the minimum distance threshold is less than the minimum distance threshold (“yes” in 420), the process 400 may proceed to block 424 and may determine whether the current object O c In some embodiments, the second time threshold may be twice the first time threshold. Figure 4 In , the second time threshold is represented as 2*T0. Alternatively, in some embodiments, the second time threshold may be a duration greater than the first time threshold, such as 1.2*T0, 1.5*T0, 3*T0, etc.

[0083] At 424, if process 400 determines that c If the age of the user is less than the second time threshold (“yes” in 424), the process 400 may proceed to block 426 and may update the value of the user. c For example, process 400 may update O based on the angle of arrival θ and the power vector v associated with the current frame of the input audio signal. c Conversely, at 424, if process 400 determines that c If the age of reaches or exceeds the second time threshold ("No" in 424), the process 400 can proceed to block 428 and a cluster to be updated can be selected based on an angle of arrival (generally represented herein as θ) associated with the current frame of the audio signal. For example, given a cluster within the region of interest and a cluster outside the region, the process 400 can select one of the two clusters based on the angle of arrival, e.g., the cluster closest to the angle of arrival.

[0084] After selecting a cluster, at 430, process 400 can determine whether all objects in the cluster have been traversed. At 430, if process 400 determines that not all objects in a given cluster have been traversed ("No" at 430), process 400 can proceed to block 432 and can determine whether a given object O within the cluster being traversed is related to object O. c Is the distance between them less than the minimum matching distance, usually denoted as d in this paper? O In other words, if given an object O and an object O c If the distance between the objects is less than the minimum matching distance, process 400 may determine that the object O c is similar enough to object O to be substantially the same in position and / or angle of arrival. The distance can be determined by:

[0085]

[0086] At 432, if process 400 determines that cIf the distance between the given object O in the cluster being traversed is less than the minimum matching distance (“yes” in 432), the process 400 can proceed to block 434 and the object O can be updated. Specifically, the process 400 can update the object O to have the object O c In addition, process 400 may set a "match" state variable to TRUE, thereby indicating that object O has been found. c matches, and can convert object O c The age is reset to 0.

[0087] On the contrary, in 434, if O c If the distance between the given object O exceeds the minimum match threshold, the process 400 can loop back to block 430 and select the next object from the cluster being traversed. Once all objects in the cluster have been traversed (e.g., "yes" in 430), the process 400 can proceed to block 436 and determine whether the "match" state variable has been set to true (e.g., in block 434). In other words, the process 400 can determine whether the current object O in the given cluster has been traversed. c A match is identified. If a match has been found ("yes" in block 436), process 400 may return to point A.

[0088] Conversely, if no match is found ("No" in block 436), process 400 may proceed to block 438 and may identify the target object based on the object. c To create a new object, you can also use O c Replace the lowest ranking object in the cluster. Then, process 400 can c The age of the object is set to 0. Note that the ranking of an object can represent the importance of the object, that is, how much speech is associated with the object. In other words, the ranking can indicate the level of speech activity associated with the object. Note that in block 438, process 400 can be based on object 0. c To determine whether to create a new object, or to replace the lowest ranked object in the cluster based on whether the cluster already has the maximum number of objects allowed in the cluster. After executing block 438, process 400 may return to point A.

[0089] As above combined Figure 3 As described above, in some embodiments, the gain can be determined based on a similarity index. In addition, objects can be retained in or removed from a cluster based on their ranking, where the ranking indicates an activity level relative to the speech activity of the object. In some embodiments, the ranking can be determined based on a similarity index for a given frequency band of a given cluster. For example, within a given cluster, the highest ranked object can be the object with the highest ranking relative to the current object. cThe most active object. For a given frequency band, the object in the cluster is closest to (e.g., has the smallest similarity index) the O of that frequency band. c In some embodiments, the most active object in a given cluster is the object with the largest number of active frequency bands.

[0090] Figure 5A 810 is a flow chart of an example process 500 for determining a similarity index and updating an object ranking according to some embodiments. In some implementations, the blocks of process 500 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a controller or processor associated with an audio or video conferencing system, etc.). An example of such a controller or processor is control system 810, such as in Figure 8 In some embodiments, the blocks of process 500 may be executed in a manner different from that of Figure 5A In some implementations, two or more blocks of process 500 may be performed substantially in parallel. In some implementations, one or more blocks of process 500 may be omitted.

[0091] Process 500 may begin at 502 by determining a similarity index for each cluster in a set of clusters and for each frequency band. As described above, in some embodiments, the set of clusters may include clusters within the region of interest and clusters outside the region of interest. In other embodiments, the set of clusters may include three, four, five, ten, etc. clusters. For a given frequency band b, the similarity index for cluster i may be set to the maximum vector inner product of the power vectors of cluster i and frequency band b and the power vector of the object within cluster i that maximizes the inner product. For example, the similarity index for frequency band b and cluster i may be determined by the following formula:

[0092]

[0093] To utilize the equations given above, for a given cluster i, process 500 can iterate through the objects assigned to cluster i and either retain the previous value of the similarity index or update the value of the similarity index based on the inner product value of the objects in the current loop iteration. An example process for iterating through objects to determine the similarity index is shown in Figure 5B FIG. 1 is shown in FIG. 1 and described below in conjunction with the figure.

[0094] At 504, process 500 may update the ranking of each object in each cluster. As described above, the ranking may indicate how active the object is relative to voice activity. The ranking of a given object may reflect the degree to which the object contributes to the similarity index of the cluster assigned to a given frequency band b. For example, in the case where the similarity index of a given cluster i and frequency band b is equal to the inner product of the power vector of frequency band b of the object and the power vector of frequency band b, the ranking of the object may be increased (e.g., increased by one, increased by two, etc.). Conversely, in the case where the similarity index of a given cluster i and frequency band b is not equal to the inner product of the power vector of frequency band b of the object and the power vector of frequency band b, the ranking of the object may be decreased (e.g., decreased by five, decreased by ten, decreased by twenty, etc.). In this way, the object that contributes most to the similarity index of a given cluster may have the highest ranking value. Recall that, as described above in conjunction with Figure 4 As described, the object with the lowest ranking may be replaced while allowing the cluster to have the maximum number of objects. Thus, the object that contributes the least to the similarity index of the cluster may be replaced, while the object that contributes more to the similarity index may be kept in the cluster.

[0095] Figure 5B A flow diagram of an example process 520 for determining similarity values ​​for clusters in a set of clusters on a per frequency band basis in accordance with some embodiments is illustrated. In some implementations, the blocks of process 520 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a computer or processor associated with an audio or video conferencing system, etc.). In some embodiments, the blocks of process 520 may be performed in a manner different from Figure 5B In some embodiments, two or more blocks of process 520 may be executed substantially in parallel. In some embodiments, one or more blocks of process 520 may be omitted. It should be noted that in some embodiments, due to the determination of the current object (generally represented as 0 c ) is greater than the minimum age, process 520 may begin at 522. For example, process 520 may correspond to Figure 4 In other words, in some embodiments, process 520 may be responsive to determining that the current object O c Age is above the minimum age (e.g. Figure 4 424).

[0096] Process 520 may begin at box 522 by setting index i for traversing a set of clusters for a given frequency band b. Note that process 520 may iterate over multiple frequency bands in the set of frequency bands. In 524, process 500 may determine whether i is less than the total number of clusters (i.e., determine whether all clusters have been traversed). In 524, if process 500 determines that i is less than the total number of clusters ("yes" in 524), process 520 may proceed to box 526 and may set index k to 0, where k is the index for traversing objects in cluster i. In 526, process 520 may initialize the similarity index of frequency band b and cluster i to 0. That is, process 520 may initialize c i,b Set to 0. At 528, process 520 may determine whether k is less than the number of objects in cluster i (i.e., determine whether all objects in cluster i have been traversed). At 528, if process 520 determines that k is less than the number of objects in cluster i (“yes” at 528), process 520 may proceed to 530 and may set the similarity index c i,b Set to the maximum of the following: 1)c i,b The current value of; and 2) the power vector of band b (denoted as v b ) and the power vector of the current object k in band b (denoted as v b k ). Process 520 may then increment k in 532. Process 520 may iterate through the objects until k reaches or exceeds the number of objects in cluster i (“no” in 528). In response to determining that all objects in cluster i have been traversed, process 520 may increment i in box 534 to advance to the next cluster. Process 520 may iterate through all clusters until it is determined that index i reaches or exceeds the number of clusters (“no” in 524). Once all clusters have been traversed (“no” in 524), process 520 may loop back to box 522 and traverse another frequency band b.

[0097] Figure 5C 5 is a flow chart of an example process 550 for updating an object ranking according to some embodiments. In some implementations, the blocks of process 550 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a computer or processor associated with an audio or video conferencing system, etc.). In some embodiments, the blocks of process 550 may be executed in a manner different from Figure 5C In some implementations, two or more blocks of process 550 may be performed substantially in parallel. In some implementations, one or more blocks of process 550 may be omitted.

[0098] Process 550 may begin at block 552 by setting index i to 0, which is used to traverse a set of clusters. In 554, process 550 may determine whether all clusters have been traversed by determining whether i is less than the number of clusters. If i is less than the number of clusters ("yes" in 554), process 550 may proceed to block 556 and may set index k to 0, which is used to traverse objects in cluster i. In 558, process 550 may determine whether all objects in cluster i have been traversed by determining whether k is less than the number of objects in cluster i. In 558, if process 550 determines that not all objects have been traversed ("yes" in 558), process 550 may proceed to block 560 and may set index b to 0, which is used to traverse all frequency bands. In 562, process 550 may determine whether all frequency bands in the set of frequency bands have been traversed by determining whether b is less than the number of frequency bands. At 562, if the process 550 determines that not all frequency bands have been traversed (“yes” at 562), the process 550 may proceed to block 564. At 564, the process 550 may determine the similarity index (given by c i,b Indicates whether the similarity index of cluster i and band b is equal to the inner product of the power vector of band b and the power vector of object k in cluster i at band b. If the similarity index of cluster i and band b is equal to the inner product of the power vector of band b and the power vector of object k in cluster i at band b ("yes" in 564), process 550 can increase the ranking of object k in cluster i in box 568. For example, the ranking can be increased by one, two, three, five, etc. On the contrary, if the similarity index of cluster i and band b is not equal to the inner product of the power vector of band b and the power vector of object k in cluster i at band b ("no" in 564), process 550 can reduce the ranking of object k in box 572. For example, the ranking can be reduced by five, ten, twenty, etc. Process 550 can then increment index b in 570 and advance to the next band. Process 550 can then traverse all bands until it is determined that all bands have been traversed ("no" in box 562). After traversing all bands, process 550 may increment object index k to continue traversing objects in cluster i at 574. Once it is determined that all objects in cluster i have been traversed ("No" in block 558), process 550 may increment index i to traverse the next cluster at 576. Once it is determined that all clusters have been traversed ("No" in block 554), process 550 may end.

[0099] As above combined Figure 2As described above, in some embodiments, the gain may be applied by a beamforming block. The beamforming block may combine the input audio signal (which may be in the frequency domain), the sub-band information, and the upper and / or lower gain limits (e.g., as determined by the audio scene analysis block and as described above in conjunction with Figure 3 The beamforming block may use a linear filter as input. In some embodiments, the beamforming block may use a linear filter. In other embodiments, the beamforming block may determine a ratio of spatial components, such as a ratio of an X component of the input audio signal to a Y component of the input audio signal, and apply a gain based on the ratio of the spatial components.

[0100] Fig. 6A is a schematic diagram of a beamforming block 600 using a linear filter. The beamforming block 600 may be used in the system 200, such as in Figure 2 For example, the beamforming block 600 may correspond to Figure 2 The beamforming block 214 is shown in FIG. Fig. 6A As shown, the beamforming block 600 may include a static beamforming block 602. The static beamforming block 602 may be configured to apply a linear filter to the input audio signal to generate a filtered audio signal. The input audio signal and the filtered audio signal may be grouped into a plurality of frequency bands via, for example, a band splitting block 604 and / or 606. Note that any suitable number of band splitting blocks may be utilized, although in the example of FIG. Fig. 6A Only two band division blocks are illustrated in FIG. Multiple gains can be determined by taking the difference between the power of the input audio signal and the filtered signal. Gain limits for each cluster (e.g., determined by the scene analysis block) can be provided to the clamp block 608. Given two clusters corresponding to a cluster within the region of interest and a cluster outside the region of interest, the gain limits can be represented by g0 and g1, where g0 represents the lower limit of the cluster within the region of interest and where g1 represents the upper limit of the cluster outside the region of interest. The gain determined by the power difference can be clamped by the clamp block 608 using the gain limits so that the clamped gain follows the gain limits. The clamped gain for each frequency band is represented as g b , and may be smoothed by a smoothing block 610 (e.g., over time or over frames of the audio signal). The smoothing gain may then be applied to the audio signal on a per-band basis to generate a modified audio signal. The modified audio signal may be in the frequency domain and may be converted to the time domain to generate an enhanced audio signal.

[0101] Figure 6B650 according to some embodiments, the beamforming block uses the ratio of the spatial components of the input audio signal to determine the gain. As shown, the X / Y ratio block 652 can take the W, X, and Y spatial components of the input audio signal as input. The spatial components can be obtained from the high-fidelity stereo conversion block, such as Figure 2 The gain engine 654 can then determine a first pass beamforming gain value based on the spatial component, generally denoted herein as g b p For example, in some embodiments, the first pass beamforming gain value may be determined by the following formula:

[0102]

[0103] G heavy , G mid and G light Example values ​​of include {-40dB, -20dB, -10dB}, {-50dB, -30dB, -20dB}, {-50dB, -30dB, -10dB}, etc.

[0104] Clamp block 656 can be g b p and gain limits for each cluster as input. For example, given two clusters corresponding to a cluster within the region of interest and a cluster outside the region of interest, the gain limits may be represented by g0 and g1, where g0 represents the lower limit for the cluster within the region of interest, and where g1 represents the upper limit for the cluster outside the region of interest. The clamp block 656 may adjust the gain of the first-pass beamforming by clamping the gain of the first-pass beamforming by g according to the gain limits. b p To generate the gain of each frequency band (denoted as g b ). In one example, the gain of a given frequency band b may be determined by:

[0105]

[0106] In the equation given above, g 0,b represents the lower limit of the gain of the cluster within the region of interest, and g 1,b Indicates the upper limit of gain for clusters outside the region of interest.

[0107] Similar to the above combination Fig. 6A As described, the gain may be smoothed by a smoothing block 658. The smoothed gain may then be applied to the input audio signal on a per frequency band basis to generate a modified audio signal. The modified audio signal may then be transformed to the time domain to generate an enhanced audio signal.

[0108] Figure 7 is a flow chart of an example process for generating an enhanced audio signal according to some embodiments. In some embodiments, the blocks of process 700 may be performed by a processor or controller, such as a processor or controller of a user device (e.g., a laptop computer, a desktop computer, a controller or processor associated with an audio or video conferencing system, etc.). An example of such a controller or processor is control system 810, such as in Figure 8 In some embodiments, the blocks of process 700 may be executed in a manner different from that of Figure 7 In some implementations, two or more blocks of process 700 may be performed substantially in parallel. In some implementations, one or more blocks of process 700 may be omitted.

[0109] Process 700 may begin at 702, i.e., receiving an input audio signal from a plurality of microphones. The number of microphones may be two, three, five, etc. The microphones may be associated with an audio conferencing system or a video conferencing system. In some implementations, a representation of the input audio signal may be transformed from a time domain to a frequency domain.

[0110] At 704, process 700 can identify an angle of arrival associated with the input audio signal. The angle of arrival can be identified relative to a particular frame of the input audio signal. In some implementations, the angle of arrival can be identified based on an A-grade stereo representation of the input audio signal (e.g., a first-order A-grade stereo representation) or based on any other suitable spatial component representation of the input audio signal. Example techniques for determining the angle of arrival are described above in conjunction with Figure 3 Described.

[0111] At 706, process 700 may determine a plurality of gains corresponding to a plurality of frequency bands of the input audio signal based on a combination of at least: 1) a representation of covariance associated with a microphone in the plurality of microphones on a per frequency band basis; and 2) an angle of arrival. The representation of covariance associated with the microphone may be a power vector. Example techniques for determining a power vector are described above in conjunction with Figure 3 Note that the gain is determined on a per-band basis. In some embodiments, the gain may be determined based on one or more gain limits, such as Figure 3 For example, a first gain limit may be applied to the region of interest and a second gain limit may be applied outside the region of interest, thereby ensuring that audio objects are not overly suppressed when within the region of interest and that audio objects outside the region of interest are sufficiently suppressed.

[0112] At 708, process 700 may apply a plurality of gains to a plurality of frequency bands of the input audio signal such that at least a portion of the input audio signal is suppressed to form an enhanced audio signal. Note that the gains are applied on a per-frequency band basis, thereby allowing for more robust suppression of competing speech outside of the region of interest, for example. In some embodiments, the gains may be smoothed prior to application, such as in Fig. 6A and 6B Note that after applying the gain to form the modified audio signal, the modified audio signal can be transformed from the frequency domain to the time domain (e.g., using an inverse STFT, as in Figure 2 ) to generate an enhanced audio signal.

[0113] In some implementations, the enhanced audio signal can be presented (eg, via a pair of loudspeakers, via a pair of headphones, etc.) In some implementations, the enhanced audio signal can be stored, for example, in a memory of the device for later playback.

[0114] Figure 8 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. Figure 1 Sample, Figure 8 The types and quantities of the elements shown in are provided as examples only. Other embodiments may include more, less and / or different types and quantities of elements. According to some examples, device 800 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 800 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker or another type of device.

[0115] According to some alternative embodiments, the apparatus 800 may be or may include a server. In some such examples, the apparatus 800 may be or may include an encoder. Thus, in some instances, the apparatus 800 may be a device configured for use within an audio environment such as a home audio environment, whereas in other instances, the apparatus 700 may be a device configured for use in the "cloud", e.g., a server.

[0116] In this example, the device 800 includes an interface system 805 and a control system 810. In some embodiments, the interface system 805 can be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 805 can be configured to exchange control information and associated data with the audio devices of the audio environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 800.

[0117] In some embodiments, the interface system 805 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. The audio data may include but may not be limited to an audio signal. In some cases, the audio data may include spatial data such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0118] The interface system 805 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 805 may include one or more wireless interfaces. The interface system 805 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 805 may include a control system 810 and a memory system (e.g., Figure 8 815). However, in some cases, the control system 810 may include a memory system. In some implementations, the interface system 805 may be configured to receive input from one or more microphones in the environment.

[0119] For example, control system 810 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0120] In some embodiments, the control system 810 may reside in more than one device. For example, in some embodiments, a portion of the control system 810 may reside in a device within one of the environments described herein, and another portion of the control system 810 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smart phone or tablet computer), etc. In other examples, a portion of the control system 810 may reside in a device within an environment, and another portion of the control system 810 may reside in one or more other devices of the environment. For example, a portion of the control system 810 may reside in a device (e.g., a server) that implements a cloud-based service, and another portion of the control system 810 may reside in another device (e.g., another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 805 may also reside in more than one device.

[0121] In some implementations, the control system 810 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 810 may be configured to implement methods of enhancing speech of interest and / or suppressing interfering noise, etc.

[0122] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 8 810 and / or the control system 810 shown in FIG. 815. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may, for example, perform scene analysis, determine gain limits for different clusters, determine gains for different frequency bands, apply gains to audio signals to generate modified or enhanced audio signals, etc. For example, the software may be implemented by a processor such as a processor. Figure 8 The control system 810 and other control system components are executed.

[0123] In some examples, apparatus 800 may include Figure 8Optional microphone system 820 is shown in . Optional microphone system 820 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 800 may not include microphone system 820. However, in some such embodiments, device 800 may still be configured to receive microphone data from one or more microphones in the audio environment via interface system 810. In some such embodiments, a cloud-based implementation of device 800 may be configured to receive microphone data or noise indicators corresponding at least in part to the microphone data from one or more microphones in the audio environment via interface system 810.

[0124] According to some embodiments, the apparatus 800 may include Figure 8 Optional loudspeaker system 825 is shown in FIG. Optional loudspeaker system 825 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 800 may not include loudspeaker system 825. In some implementations, device 800 may include headphones. The headphones may be connected or coupled to device 800 via a headphone jack or via a wireless connection (e.g., Bluetooth).

[0125] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0126] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP), which is configured (e.g., programmed and otherwise configured) to perform the required processing on (multiple) audio signals, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of various operations, including one or more examples of the disclosed method. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0127] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for executing one or more examples of the disclosed method or its steps (e.g., an encoder executable to execute one or more examples of the disclosed method or its steps).

[0128] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method for processing audio, the method comprising: receiving input audio signals from a plurality of microphones; identifying an angle of arrival associated with the input audio signal; determining a plurality of gains corresponding to a plurality of frequency bands of the input audio signal based on a combination of at least: 1) a representation of the covariance of a signal associated with a microphone of the plurality of microphones on a per frequency band basis; and 2) the angle of arrival; as well as The plurality of gains are applied to the plurality of frequency bands of the input audio signal such that at least a portion of the input audio signal is suppressed to form an enhanced audio signal.

2. The method of claim 1, wherein: Identifying the angle of arrival includes converting the received signal associated with a microphone of the plurality of microphones into a spatial representation, and wherein the input audio signal corresponds to the spatial representation.

3. The method according to any one of claims 1 to 2, wherein: Determining the plurality of gains comprises: identifying one or more objects of the input audio signal; and The one or more objects of the input audio signal are clustered within one of a plurality of clusters, wherein the plurality of gains associated with a current time frame of the input audio signal are determined based on proximity of the current time frame of the input audio signal to objects within the clusters of the one or more objects.

4. The method of claim 3, wherein: Identifying the one or more objects of the input audio signal is based on current input and historical input.

5. The method according to any one of claims 3 or 4, wherein: Clustering the one or more objects of the input audio signal is responsive to determining that a number of frames in which the one or more audio objects appear in the input audio signal has exceeded a threshold number of frames.

6. The method according to any one of claims 3 to 5, wherein: Clustering a given object of the one or more objects of the input audio signal comprises one of: 1) updating an existing object in a cluster; 2) creating a new object in the cluster corresponding to the given object; or 3) replacing the existing object in the cluster with the given object.

7. The method of claim 6, wherein: The existing object that is replaced is the existing object with the lowest activity level in the cluster.

8. The method according to any one of claims 3 to 7, wherein: The clustering is performed on a broadband basis for the plurality of frequency bands.

9. The method according to any one of claims 3 to 8, wherein: Clustering the one or more objects includes determining a plurality of similarity indicators between the input audio signal and each cluster.

10. The method of claim 9, wherein: The plurality of similarity indicators correspond to the plurality of frequency bands.

11. The method according to any one of claims 9 or 10, wherein: Determining a similarity index for a given cluster is based on the most active objects within the given cluster.

12. The method according to any one of claims 9 to 10, wherein: The plurality of gains are determined using the plurality of similarity indicators.

13. The method according to any one of claims 3 to 12, wherein: The plurality of clusters include clusters within a region of interest and clusters outside the region of interest.

14. The method of claim 13, further comprising determining, for each of the plurality of frequency bands, a lower gain limit applicable to a portion of the input audio signal inside the region of interest and an upper gain limit applicable to a portion of the input audio signal outside the region of interest, wherein The plurality of gains are limited by the lower gain limit and the upper gain limit.

15. The method according to any one of claims 1 to 13, wherein: Applying the plurality of gains comprises: filtering the input audio signal using a linear filter to generate a filtered signal; grouping the input audio signal and the filtered signal into the plurality of frequency bands; calculating the plurality of gains for the plurality of frequency bands by taking the difference between the power of the input audio signal and the filtered signal; determining multiple gain bounds; clamping the gain to the gain limit; and A clamped gain is applied to the input audio signal.

16. The method according to any one of claims 1 to 15, wherein: Applying the plurality of gains comprises: determining a ratio of spatial components of the input audio signal; and The plurality of gains are applied based at least in part on the ratio of the spatial components.

17. The method of any one of claims 1 to 16, further comprising smoothing the plurality of gains before applying the plurality of gains.

18. The method of claim 17, further comprising causing the enhanced audio signal to be presented via a loudspeaker or headphones.

19. A system comprising one or more processors configured to perform the operations of any one of claims 1 to 18.

20. A computer program product configured to cause one or more processors to perform the operations of any one of claims 1 to 18.

Images