Multi-channel noise reduction for headphones

Abstract

The invention provides a method, a device and a system for audio signal processing. The present embodiments more specifically relate to speech enhancement techniques that can adapt to varying signal-to-noise ratio (SNR) conditions. In some aspects, a speech enhancement system may include a low SNR detector and a spatial filter. The spatial filter receives the multi-channel audio signal via the microphone array and generates an enhanced audio signal based on the beamforming filter. The low SNR detector tracks the SNR of a reference audio signal of the multi-channel audio signal. In some implementations, when the SNR falls below a broadband SNR threshold level, the spatial filter may replace at least a portion of the reference audio signal with a secondary audio signal received from a secondary microphone separate from the microphone array. In some other embodiments, the spatial filter may avoid updating the beamforming filter when the SNR falls below a narrowband SNR threshold level.

Description

Technical Field

[0001] The present embodiments relate generally to signal processing, and in particular to multi-channel noise reduction techniques for headphones. Background Art

[0002] Many hands-free communication devices include a microphone configured to convert sound waves into an audio signal that can be transmitted to a receiving device via a communication channel. An audio signal typically includes a speech component (such as from a user of the communication device) and a noise component (such as from a reverberant enclosure). Speech enhancement is a signal processing technique that attempts to suppress the noise component of a received audio signal without distorting the speech component. Many existing speech enhancement techniques rely on statistical signal processing algorithms that continuously track the noise pattern in each frame of the audio signal to model a spectral suppression gain or filter that can be applied to the received audio signal in the time-frequency domain.

[0003] Beamforming is a signal processing technique that can focus the energy of an audio signal in a specific spatial direction. More specifically, a beamformer can improve the speech quality in an audio signal received via a microphone array by combining the signals at the microphone outputs. For example, a beamformer can apply corresponding weights to the audio signals output by each microphone in the array so that when the audio signals are combined, the signal strength is enhanced in the direction of the speech (or suppressed in the direction of the noise). An adaptive beamformer is able to dynamically adjust the weights applied to the microphone outputs to optimize the quality or signal-to-noise ratio (SNR) of the combined audio signal. Example adaptive beamforming techniques include, among other examples, minimum mean square error (MMSE), minimum variance distortionless response (MVDR), generalized eigenvalue (GEV), and generalized sidelobe cancellation (GSC).

[0004] In a low SNR environment, an adaptive beamformer may converge in a direction different from the speech direction, such as the direction of a dominant noise source. As a result, the adaptive beamformer may distort or even suppress speech components of an audio signal with a low SNR. Therefore, there is a need to prevent an adaptive beamformer from converging in a wrong direction under low SNR conditions. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] An innovative aspect of the subject matter of the present disclosure can be implemented in a method for speech enhancement. The method includes: receiving a plurality of audio signals via a plurality of microphones of a microphone array, respectively, wherein each of the plurality of audio signals represents a corresponding channel of a multi-channel audio signal; receiving an auxiliary audio signal via an auxiliary microphone separated from the microphone array; detecting a wideband signal-to-noise ratio (SNR) of a reference audio signal of the plurality of audio signals; selectively replacing at least a portion of the reference audio signal with the auxiliary audio signal based on the wideband SNR, so that as a result of the replacement, the multi-channel audio signal includes the auxiliary audio signal to replace the at least a portion of the reference audio signal; and enhancing the speech component of the multi-channel audio signal based on a minimum variance distortionless response (MVDR) beamforming filter.

[0007] Another innovative aspect of the subject matter of the present disclosure can be implemented in a speech enhancement system, which includes a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the speech enhancement system to: receive multiple audio signals via multiple microphones of a microphone array, respectively, wherein each of the multiple audio signals represents a corresponding channel of a multi-channel audio signal; receive an auxiliary audio signal via an auxiliary microphone separate from the microphone array; detect a wideband SNR of a reference audio signal of the multiple audio signals; selectively replace at least a portion of the reference audio signal with the auxiliary audio signal based on the wideband SNR, so that as a result of the replacement, the multi-channel audio signal includes the auxiliary audio signal to replace the at least a portion of the reference audio signal; and enhance the speech component of the multi-channel audio signal based on an MVDR beamforming filter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present embodiments are shown by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0009] Figure 1 An example environment is shown for which speech enhancement may be implemented.

[0010] Figure 2 An example audio receiver supporting multi-channel beamforming is shown.

[0011] Figure 3 A block diagram of an example speech enhancement system is shown in accordance with some implementations.

[0012] Figure 4 A block diagram of an example low signal-to-noise ratio (SNR) detection system is shown in accordance with some implementations.

[0013] Figure 5A A block diagram of an example narrowband SNR detection system is shown in accordance with some embodiments.

[0014] Figure 5B A block diagram of an example wideband SNR detection system is shown in accordance with some implementations.

[0015] Figure 6 A block diagram of an example adaptive beamforming system is shown in accordance with some implementations.

[0016] Figure 7 Another block diagram of an example speech enhancement system is shown in accordance with some implementations.

[0017] Figure 8 An illustrative flow chart depicting example operations for speech enhancement is shown in accordance with some implementations. DETAILED DESCRIPTION

[0018] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" means being directly connected to or connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" can be used interchangeably to refer to any system that can electronically process information. Moreover, in the following description and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure difficult to understand. The subsequent parts of the detailed description are presented in terms of processes, logic blocks, processing, and other symbolic representations of the operation of data bits in a computer memory.

[0019] These descriptions and representations are the means used by the technicians in the data processing field to most effectively convey the essence of their work to other technicians in the field. In the present disclosure, processes, logic blocks, processes, etc. are considered to be the self-consistent sequences of steps or instructions that lead to desired results. Steps are those steps that require the physical operation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise operated in a computer system. However, it should be remembered that all of these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0020] Unless otherwise specifically stated, as will be apparent from the following discussion, it is appreciated that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system or similar electronic computing device to manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0021] In the accompanying drawings, a single block may be described as implementing one or more functions; however, in actual practice, one or more functions implemented by the block may be implemented in a single component or across multiple components, and / or may be implemented using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described below in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present disclosure. Similarly, the example input device may include components different from those shown, including known components such as processors, memories, etc.

[0022] Unless the techniques described herein are specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed, implement one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0023] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0024] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors (or processing systems). The term "processor" as used herein may refer to any general purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0025] As described above, beamforming is a signal processing technique that can focus the energy of an audio signal (also referred to as a "multi-channel audio signal") received via a microphone array in a specific spatial direction. For example, an adaptive minimum variance distortionless response (MVDR) beamformer can determine a set of weights (also referred to as an "MVDR beamforming filter") that reduces or minimizes the noise component of the multi-channel audio signal without distorting the speech component. More specifically, the MVDR beamforming filter coefficients can be determined as a function of the covariance of the noise component of the multi-channel audio signal and a set of relative transfer functions (RTFs) between the microphones of the microphone array (also referred to as "RTF vectors"). However, when the signal-to-noise ratio (SNR) of the audio signal is low, the adaptive MVDR beamformer may converge in a direction different from the speech direction (such as the direction of the dominant noise source), which can result in greater speech distortion.

[0026] Aspects of the present disclosure recognize that, for some audio receivers, the positioning of the microphone array can be relatively fixed relative to the target audio source. For example, when a headset is worn by any user (or "speaker"), the microphone installed on the headset can detect speech from substantially the same direction. As such, the RTF vector associated with the microphone array installed on the headset should present very small (if any) changes over time. Aspects of the present disclosure also recognize that many headsets have auxiliary microphones that isolate noise better than the microphones of the microphone array. In addition to other examples, the example auxiliary microphone may include a bone conduction microphone (which detects speech based on vibrations in the user's skull) and an internal microphone (which may be located in the ear cup of the headset and is typically used to provide feedback for active noise cancellation (ANC) systems). Therefore, the audio signal received via the auxiliary microphone can be used to supplement the audio signal received via the microphone array under low SNR conditions.

[0027] Various aspects generally relate to audio signal processing, and more particularly, to speech enhancement techniques that can be adapted to changing SNR conditions. In some aspects, a speech enhancement system may include a low SNR detector and a spatial filter. The spatial filter is configured to receive a multi-channel audio signal via a microphone array and generate an enhanced audio signal based on an MVDR beamforming filter. In some embodiments, the spatial filter may determine the MVDR beamforming filter based at least in part on an RTF vector associated with the microphone array. The low SNR detector is configured to track the SNR of a reference audio signal of the multi-channel audio signal. In some embodiments, when the SNR drops below a broadband SNR threshold level, the spatial filter may replace at least part of the reference audio signal with an auxiliary audio signal, wherein the auxiliary audio signal is received via an auxiliary microphone (such as a bone conduction microphone or an internal microphone) separated from the microphone array. In some other embodiments, when the SNR drops below a narrowband SNR threshold level, the spatial filter may avoid updating the RTF vector.

[0028] Specific embodiments of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages. By replacing at least a portion of a reference audio signal with an auxiliary audio signal, aspects of the present disclosure may improve speech quality in a multi-channel audio signal through MVDR beamforming even under low SNR conditions. For example, since the auxiliary microphone isolates noise better than the microphone of the microphone array, the auxiliary audio signal may have a significantly higher SNR than the reference audio signal. Therefore, replacing the reference audio signal with the auxiliary audio signal may improve the SNR of the multi-channel audio signal. By avoiding updating the RTF vector under low SNR conditions, aspects of the present disclosure may prevent the MVDR beamforming filter from converging in the wrong direction. For example, the MVDR beamforming filter may be locked to a predetermined RTF vector that is known to result in a relatively accurate beam direction. As such, the MVDR beamforming filter cannot be adapted to the direction of the dominant noise source.

[0029] Figure 1 An example environment 100 is shown for which speech enhancement may be implemented. Example environment 100 includes headset 110 and user 120. In some aspects, headset 110 may include an array of microphones 112 and 114. Figure 1 In the example of FIG. 1 , the microphone array is shown to include only two microphones 112 and 114. However, in actual implementations, the microphone array may include more than Figure 1 There are more microphones than those pictured in the video.

[0030] Microphones 112 and 114 are positioned or otherwise configured to detect speech 122 (depicted as a series of sound waves) propagating from the mouth of user 120. For example, each of microphones 112 and 114 can convert the detected speech 122 into an electrical signal (also referred to as an "audio signal") representing the sound waveform. Each audio signal can include a speech component (representing the user's speech 122) and a noise component (representing background noise from the headset 110 or the surrounding environment). Due to the spatial positioning of microphones 112 and 114, speech 122 detected by some microphones in the microphone array can be delayed relative to speech 122 detected by some other microphones in the microphone array. In other words, microphones 112 and 114 can produce audio signals with different phase offsets.

[0031] In some aspects, the audio signals generated by each of the microphones 112 and 114 of the microphone array may be weighted and combined to enhance the speech component of the audio signal or suppress the noise component. More specifically, the weight applied to the audio signal may be configured to improve the signal strength in the direction of the speech 122. Such signal processing techniques are generally referred to as "beamforming". In some embodiments, an adaptive beamformer may estimate (or predict) a set of weights (also referred to as a "beamforming filter") to be applied to the audio signal, which enhances the signal strength in the direction of the speech. The speech quality in the generated signal depends on the accuracy of the beamforming filter coefficients. For example, when the beamforming filter is aligned in the direction of the user's mouth, the speech may be enhanced. On the other hand, if the beamforming filter is aligned in the direction of the noise source, the speech is distorted or suppressed.

[0032] An adaptive beamformer can dynamically adjust the beamforming filter coefficients to optimize the quality or signal-to-noise ratio (SNR) of the combined audio signal. For example, a minimum variance distortionless response (MVDR) beamformer can determine a beamforming filter that reduces or minimizes the noise component of an audio signal without distorting the speech component. MVDR beamforming assumes that only a delay-only propagation path exists between the microphones 112 and 114 of the microphone array and the audio source. However, in a headset-mounted configuration, the audio signal generated by the microphones 112 and 114 may include acoustic background noise from the housing or reverberant housing of the headset 110. When the SNR of the audio signal is too low, the phase information of the speech component may be corrupted by the dominant noise source. As a result, the MVDR beamforming filter may converge in directions other than the speech direction (such as the direction of the dominant noise source), which may result in significant speech distortion or cancellation.

[0033] In some embodiments, the headset 110 may also include an auxiliary microphone 116 that is separate from the microphone array. More specifically, the auxiliary microphone 116 may be better at isolating noise than any of the microphones 112 or 114 of the microphone array. For example, Figure 1 As shown in , microphones 112 and 114 of the microphone array are disposed on an outer surface of the housing of the headset 110, while an auxiliary microphone 116 is disposed on an inner surface of the housing that is closer to the user 120 than the outer surface. Example suitable auxiliary microphones may include, among other examples, a bone conduction microphone (which detects speech based on vibrations in the user's skull) and an internal microphone (which may be located in an ear cup of the headset 110 and used to provide feedback for an active noise cancellation (ANC) system). Figure 1In the example of FIG. 1 , the headset 110 is shown as including a single auxiliary microphone 116. However, in actual implementations, the headset 110 may include any number of auxiliary microphones.

[0034] The auxiliary microphone 116 may not be able to detect as wide a range of audio frequencies as the microphones 112 and 114 of the microphone array. For example, a bone conduction microphone may be adapted to detect audio frequencies below 800 Hz, while an internal microphone may be adapted to detect audio frequencies in the range of 800 Hz to 2.5 KHz. However, due to the positioning of the auxiliary microphone 116 (such as in an ear cup) or the technology used by the auxiliary microphone 116 to detect speech (such as an accelerometer), the audio signals received via the auxiliary microphone 116 (also referred to as "auxiliary audio signals") may have a higher SNR than the audio signals received via the microphones 112 and 114 of the microphone array. Therefore, in some aspects, for the purpose of beamforming in a low SNR environment (such as when the SNR of the audio signals received via the microphone array is below a threshold level), the headset 110 may supplement or replace one or more audio signals received via the microphone array with one or more auxiliary audio signals, respectively.

[0035] Figure 2 An example audio receiver 200 supporting multi-channel beamforming is shown. The audio receiver 200 includes a number (M) of microphones 210 (1)-210 (M) arranged in a microphone array and a beamforming filter 220. In some embodiments, the audio receiver 200 may be Figure 1 For example, refer to Figure 1 , each of microphones 210 ( 1 )- 210 (M) may be an example of any microphone in microphones 112 or 114 .

[0036] The microphones 210(1)-210(M) are configured to convert a series of sound waves 201 (also referred to as “sound waves”) into audio signals X, respectively. 1 (l,k)–X M (l,k), where l is the frame index and k is the frequency index associated with the time-frequency domain. Figure 2 As shown in FIG. 1 , sound wave 201 is incident on microphones 210 ( 1 )- 210 (M) at an angle (θ). Angle θ may also be referred to as audio signal X 1 (l, k)-X M In some embodiments, sound waves 201 may include user speech (such as a user's voice) mixed with noise or interference (such as from a reverberant enclosure). Figure 1 The target speech and the distractor speech represent the audio signal X1 (l, k)-X M The speech component (S(l,k)) and the noise component (N(l,k)) in each audio signal in (l,k).

[0037] Due to the spatial positioning of microphones 210(1)-210(M), the audio signal X 1 (l, k)-X M Each audio signal in (l, k) may represent a delayed version of the same audio signal. 1 (l, k) is used as the reference audio signal, and the remaining audio signal X 2 (l, k)-X M Each of (l, k) can be described as the first audio signal X 1 (l, k) is a phase-delayed version of the audio signal X. 1 (l, k)-X M (l, k) can be modeled as a vector (X(l, k)): X(l,k)=a(θ,k)S(l,k)+N(l,k) (1) where X(l, k) = [X 1 (l, k), ..., X M (l, k)] T is a multi-channel audio signal and a(θ, k) is a steering vector representing a set of phase delays of the sound wave 201 incident on the microphones 210(1)-210(M).

[0038] The beamforming filter 220 converts the weight vector w(l, k)=[w 1 (l, k), ..., w M (l, k)] T (W 1 -w M The filter coefficients (called filter coefficients) are applied to the audio signal X(l, k) to produce an enhanced audio signal (Y(l, k)): Y(l, k) = w H (l, k)X(l, k)=w H (l, k)a(θ, k)S(l, k)+w H (l, k)N(l, k) (2) The weight vector w(l, k) determines the direction of the "beam" associated with the beamforming filter 220. Therefore, the filter coefficients w can be adjusted 1 -w M to “steer” the beam in various directions.

[0039] In some aspects, an adaptive beamformer (not shown for simplicity) may determine a weight vector w(l, k) that optimizes the enhanced audio signal Y(l, k) with respect to one or more conditions. For example, the MVDR beamformer is configured to determine a weight vector w(l, k) that reduces or minimizes the variance of the noise component of the enhanced audio signal Y(l, k) without distorting the speech component of the enhanced audio signal Y(l, k). In other words, the weight vector w(l, k) may satisfy the following conditions: argmin w w H (l,k)Φ NN (l, k)w(l, k)stw H (l, k)a(θ, k) = 1 where Φ NN (l, k) is the covariance of the noise component N(l, k) of the received audio signal X(l, k). The generated weight vector w(l, k) is the MVDR beamforming filter (W MVDR (k)), which can be expressed as:

[0040] As shown in Equation 3, some MVDR beamformers may rely on geometry (such as the steering vector a(θ, k)) to determine the weight vector w(l, k). As such, the MVDR beamforming filter w MVDR The accuracy of (l, k) depends on the accuracy of the estimate of the steering vector a(θ, k), which may be difficult to adapt to different users. Aspects of the present disclosure recognize that the MVDR beamforming filter w MVDR (l, k) can be further expressed as the covariance (Φ) of the speech component S(l, k) of the received audio signal X(l, k) SS (l, k)) function: where u(l, k) is a one-hot vector representing the reference microphone channel, and W norm (l, k) is a normalization factor associated with W(l, k). Example suitable normalization factors include W norm (l, k) = max(|W(l, k)|) and W norm (l, k) = trace(W(l, k)).

[0041] Aspects of the present disclosure also recognize that the steering vector a(θ, k) can be expressed as a vector of relative transfer functions (RTFs) between each of the microphones 210(1)-210(M) and a reference microphone within the microphone array, such as microphone 210(1). In addition, the speech covariance Φ can be used to determine the relative transfer function (RTF) between the microphones 210(1)-210(M) and the reference microphone within the microphone array. SS (l, k) to estimate the RTF vector associated with the target speech d(l, k) = [d 1 (l, k), ..., d M (l, k)] = ΦSS(l, k)W MVDR (l, k) The RTF vector Substituting into equation 3, we get:

[0042] In some aspects, the noise covariance Φ NN (l, k) and speech covariance Φ SS (l, k) can be estimated or updated over time through supervised learning. For example, when there is speech in the received audio signal X(l, k), the speech covariance Φ can be estimated SS (l, k), and when there is no speech in the received audio signal X(l, k), the noise covariance Φ can be estimated NN (l, k). In some embodiments, a deep neural network (DNN) can be used to determine whether speech is present or absent in the audio signal X(l, k). For example, the DNN can be trained to infer the likelihood or probability of speech in each frame of the audio signal X(l, k). More specifically, the DNN can be used as a voice activity detector (VAD), or used within a voice activity detector (VAD). However, when the SNR of the audio signal X(l, k) is too low (such as below a threshold level), the phase information of the user's voice may be corrupted by the dominant noise source. As a result, existing adaptive beamformers may converge in a direction different from the direction of the speech, which may result in distortion or elimination of speech in the enhanced audio signal Y(l, k).

[0043] Figure 3 is a block diagram of an example speech enhancement system 300 according to some embodiments. The speech enhancement system 300 is configured to generate an enhanced audio signal Y(l,k) based at least in part on a multi-channel audio signal X(l,k) received via a microphone array. In some embodiments, the microphone array may be Figure 1 The microphones 112 and 114 or Figure 2An example of microphones 210(1)-210(M). Figure 3 As shown in FIG, the multi-channel audio signal X(l, k) includes a number (M) of component audio signals X 1 (l, k)-X M (l, k), each component audio signal represents a corresponding channel of the multi-channel audio signal X(l, k).

[0044] The speech enhancement system 300 includes a low SNR detector 310 and a spatial filter 320. The low SNR detector 310 is configured to generate a reference audio signal (X) based on the multi-channel audio signal X(l, k). 1 (l, k)) to detect one or more low SNR conditions. Reference audio signal X 1 (l, k) represents the audio signal received by the reference microphone of the microphone array. Figure 2 The reference microphone can be any microphone in the microphone array, which is used to calculate the RTF vector In some aspects, the low SNR detector 310 can be based on the reference audio signal X 1 (l, k) to track the SNR, and the SNR may be compared to one or more threshold SNR levels. More specifically, the low SNR detector 310 may indicate that the reference audio signal X 1 Whether the SNR of (l, k) is below one or more threshold SNR levels (as indicated by a “low SNR” signal 302).

[0045] In some embodiments, the low SNR detector 310 can track the reference audio signal X 1 The term “wideband SNR” as used herein refers to the wideband SNR of the reference audio signal X measured across all frequency bins k. 1 Therefore, the low SNR detector 310 can estimate the total SNR of the reference audio signal X 1 The single wideband SNR value (SNR) for each frame l of (l, k) wb (l)), and the low SNR signal 302 may indicate that the SNR wb (l) is lower than the broadband SNR threshold. In some other embodiments, the low SNR detector 310 can track the reference audio signal X 1 The term “narrowband SNR” as used herein refers to the narrowband SNR of the reference audio signal X measured at each frequency bin k. 1 Therefore, the low SNR detector 310 can estimate the corresponding SNR of the reference audio signal X 1(l, k) of each frame l with several (K) narrowband SNR values ​​(SNR nb (l, k)), where k∈[1, K], and the low SNR signal 302 may indicate an SNR nb Whether each value of (l, k) is below the narrowband SNR threshold.

[0046] The spatial filter 320 is configured to apply the weight vector w(l,k) to the audio signal X(l,k) to produce an enhanced audio signal Y(l,k) (such as according to Equation 2). In some embodiments, the spatial filter 320 can be an adaptive beamformer that determines the weight vector w(l,k) to be applied to each frame l of the audio signal X(l,k) based at least in part on the probability of speech associated with the corresponding audio frame (p(l,k)). For example, the probability of speech p(l,k) can be inferred by a DNN trained to detect speech in an audio signal. As shown in Equations 4-6, the covariance Φ of the noise in the audio signal X(l,k) can be calculated based on the covariance Φ of the noise in the audio signal X(l,k). NN (l, k) and the covariance Φ of the speech SS (l, k) to determine the MVDR beamforming filter w MVDR In some aspects, the spatial filter 320 may dynamically update the speech covariance Φ based on the probability p(l, k) of speech associated with the corresponding audio frame. SS (l, k) and the noise covariance Φ NN (l, k).

[0047] As reference Figure 1 and 2 As described above, when the SNR of the audio signal X(l, k) is too low, the spatial filter 320 may not be able to accurately estimate the speech covariance Φ SS (l, k). For speech covariance Φ SS The incorrect estimation of (l, k) can cause the beamforming filter W MVDR (l, k) converges in the wrong direction (such as towards the dominant noise source), which can cause speech distortion or cancellation in the enhanced audio signal Y(l, k). In some aspects, when the low SNR signal 302 indicates that the reference audio signal X 1 When the SNR of (l, k) is below a threshold SNR level, the spatial filter 320 may avoid updating the beamforming filter w MVDR For example, the spatial filter 320 may use the beamforming filter w MVDR The filter coefficients of (l, k) are locked to beam directions that are known to result in relatively accurate or stable speech enhancement. As a result, the beamforming filter w MVDR (l, k) cannot converge in the direction of the dominant noise source.

[0048] In some other aspects, the spatial filter 320 may be configured to filter the audio signal received via an auxiliary microphone (not shown for simplicity). aux (l, k)) replaces or substitutes the reference audio signal X 1 (l, k) to compensate for the reference audio signal X having a low SNR 1 For example, when the low SNR signal 302 indicates that the reference audio signal X 1 When the SNR of (l, k) is below a threshold SNR level, the spatial filter 320 may modify the multi-channel audio signal X(l, k) to include the auxiliary audio signal X aux (l, k), instead of the reference audio signal X 1 In some embodiments, the auxiliary microphone may be Figure 1 An example of an auxiliary microphone 116 (such as a bone conduction microphone or an internal microphone) is shown in FIG. Since the auxiliary microphone is better isolated from noise than any microphone in the microphone array, when the reference audio signal X 1 When the SNR of (l, k) is low, the reference audio signal X 1 At least part of (l, k) is replaced by the auxiliary audio signal X aux (l, k) can improve the speech quality in the enhanced audio signal Y(l, k).

[0049] Figure 4 4 shows a block diagram of an example low SNR detection system 400 according to some embodiments. The low SNR detection system 400 is configured to track an audio signal (X 1 The narrowband SNR (SNR nb (l, k)) and wideband SNR (SNR wb (l)), and generates a low SNR detection flag (D nb (l, k) and D wb (l)), which indicates the SNR nb (l, k) and SNR wb (l) whether it is below a corresponding threshold SNR level. In some embodiments, the low SNR detection system 400 may be Figure 3 An example of a low SNR detector 310 is shown in FIG. Figure 3 , audio signal X 1 (l, k) may be a reference audio signal of the multi-channel audio signal X(l, k), and the low SNR detection flag D nb (l, k) and D wb (l) may be included in, or otherwise indicated by, the low SNR signal 302 .

[0050] The low SNR detection system 400 includes a VAD 410, a narrowband SNR detector 420, a narrowband SNR comparator 430, a wideband converter 440, a wideband SNR detector 450, and a wideband SNR comparator 460. The VAD 410 is configured to determine or predict the audio signal X. 1 More specifically, VAD 410 generates a VAD parameter (VAD(l)) indicating whether speech is present (or not) in the audio signal X. 1 (l, k) whether speech is present or absent in the current frame l. In some embodiments, VAD 410 may include a processor trained to infer the audio signal X 1 The probability (p) of speech in (l, k) DNN (l, k)) DNN, and VAD 410 can be based on the probability p of speech DNN (l, k) generates the VAD parameters VAD(l). For example, if the probability of speech p averaged across all frequency bins k is DNN (l, k) is greater than the threshold probability, then VAD 410 can determine that 1 There is speech in (l, k) (VAD(l)=1).

[0051] In some other embodiments, VAD 410 can be based on the Figure 1 The auxiliary audio signal (such as an auxiliary microphone 116) received by Figure 3 Auxiliary audio signal X aux (l, k)) to generate the VAD parameter VAD(l). Figure 1 As described above, some auxiliary microphones (such as bone conduction microphones) are configured to capture only the energy of the user's voice. In other words, background noise may not be present or otherwise reflected in the auxiliary audio signal. Therefore, aspects of the present disclosure recognize that the energy in the auxiliary audio signal can directly indicate the presence or absence of speech. For example, if the energy of the auxiliary audio signal is greater than a threshold energy level, VAD 410 can determine that the audio signal X is present. 1 There is speech in (l, k) (VAD(l)=1).

[0052] The narrowband SNR detector 420 is configured to detect the audio signal X based on the 1 (l, k) and VAD parameters VAD(l) to estimate SNR nb In some embodiments, the narrowband SNR detector 420 can track the audio signal X based at least in part on the VAD parameter VAD(l). 1 The noise floor of (l, k) and the audio signal X 1For example, the narrowband SNR detector 420 can detect the narrowband speech energy in the audio signal X 1 When there is no speech in (l, k) (such as when VAD(l)=0), the audio signal X is estimated or updated. 1 (l, k) noise floor, and can be in the audio signal X 1 The narrowband SNR detector 420 may further estimate or update the narrowband speech energy when speech is present in (l, k) (such as when VAD(l)=1). 1 The noise floor and X 1 The narrowband speech energy in (l, k) is used to calculate the SNR nb In some embodiments, the narrowband SNR detector 420 may estimate the SNR in an equivalent rectangular bandwidth (ERB) resolution. nb (l, k).

[0053] The narrowband SNR comparator 430 converts the SNR nb (l, k) and narrowband SNR threshold (T nb ) to generate a narrowband low SNR detection flag D nb (l, k). For example, when SNR nb (l, k) is less than the narrowband SNR threshold (T nb ), the narrowband SNR comparator 430 can detect the low SNR condition (D nb (l, k) = 1). On the other hand, when SNR nb (l, k) is greater than or equal to the narrowband SNR threshold T nb When the narrowband SNR comparator 430 may not detect the low SNR condition (D nb (l, k) = 0). In some embodiments, for narrowband SNR thresholds T in different frequency ranges, nb It can be different. For example, the narrowband SNR threshold T nb (k) may vary as a function of the frequency bin k. In such an embodiment, the narrowband SNR comparator 430 may convert the SNR nb (l, k) and the narrowband SNR threshold T in the logarithmic domain nb (k) Make comparisons.

[0054] Unlike narrowband SNR, wideband SNR represents the audio signal X measured across all frequency bins k. 1 In other words, the low SNR detection system 400 can only track the audio signal X 1 The SNR of each frame l of (l, k) wb In some embodiments, the broadband converter 440 may determine the audio signal X1 The broadband energy (X1) in each frame l of (l, k) tot (l)): Where K min and K max Defines the frequency range associated with speech. In some embodiments, K min and K max Can be configured to span a frequency range detectable by a bone conduction microphone (such as 50 Hz to 80 Hz). In some other embodiments, K min and K max Can be configured to span the frequency range detectable by the internal microphone (such as 800 Hz to 1.5 kHz).

[0055] The wideband SNR detector 450 is configured to be based on the wideband energy X1 tot (l) and VAD parameters VAD(l) to estimate SNR wb In some embodiments, the wideband SNR detector 450 can track the wideband energy X1 based at least in part on the VAD parameter VAD(l). tot (l) The noise floor and X1 tot (l) The wideband speech energy in the audio signal X. 1 When there is no speech in (l, k) (such as when VAD(l) = 0), estimate or update X1 tot (l) the noise floor, and can be 1 The wideband SNR detector 450 can further estimate or update the wideband speech energy based on X1 tot (l) Noise floor and X1 tot (l) to calculate the SNR wb (l).

[0056] The wideband SNR comparator 460 converts the SNR wb (l) and the broadband SNR threshold (T wb ) to generate a broadband low SNR detection flag D wb (l). For example, when SNR wb (l) is less than the broadband SNR threshold T wb When the wideband SNR comparator 460 can detect the low SNR condition (D wb On the other hand, when SNR wb (l) Greater than or equal to the broadband SNR threshold T wbWhen the wideband SNR comparator 460 may not detect the low SNR condition (D wb (l)=0).

[0057] Figure 5A FIG. 5 is a block diagram of an example narrowband SNR detection system 500 according to some embodiments. The narrowband SNR detection system 500 is configured to detect an audio signal (X) 1 The corresponding narrowband SNR value (SNR) per frequency bin is determined for each frame l of (l, k) nb In some embodiments, the narrowband SNR detection system 500 may be Figure 4 An example of a narrowband SNR detector 420. For example, refer to Figure 3 , audio signal X 1 (l, k) may be a reference audio signal of the multi-channel audio signal X(l, k).

[0058] The narrowband SNR detection system 500 includes a noise floor updating component 502, a speech energy updating component 504, and a narrowband SNR estimation component 506. The noise floor updating component 502 is configured to estimate the audio signal X based on the VAD parameter (VAD(l)). 1 The narrowband noise floor (NF) of (l, k) nb (l, k)). For example, refer to Figure 4 , the VAD parameter VAD(l) may be generated by the VAD 410. More specifically, the VAD parameter (VAD(l)) may indicate the 1 In some embodiments, when the VAD parameter VAD(l) indicates whether speech is present or absent in each frame l of the audio signal X 1 When there is speech in (l, k), the noise floor updating component 502 can avoid updating the narrowband noise floor NF nb (l, k): NF nb (l, k) = NF nb (l-1, k) if VAD(l)=1

[0059] When the VAD parameter VAD(l) indicates that the audio signal X 1 When there is no speech in (l, k) (such as when VAD(l)=0), the noise floor updating component 502 can estimate the narrowband noise floor NF for each frequency bin k. nb In some embodiments, the noise floor updating component 502 can be based on the estimated narrowband noise floor NF nb Is (l, k) lower than the audio signal X? 1 The energy level of (l, k) will be smoothed upward by a factor (α up) or downward smoothing factor (α dn ) is applied to the narrowband noise floor update, where α up >α dn :

[0060] The speech energy updating component 504 is configured to estimate the audio signal X based on the VAD parameter VAD(l) 1 The narrowband speech energy (Ps nb In some embodiments, when the VAD parameter VAD(l) indicates that the audio signal X 1 When there is no speech in (l, k), the speech energy updating component 504 can avoid updating the narrowband speech energy Ps nb (l, k): Ps nb (l, k) = Ps nb (l-1, k) if VAD(l)=0

[0061] When the VAD parameter VAD(l) indicates that 1 When speech is present in (l, k) (such as when VAD(l)=1), the speech energy update component 504 can estimate the narrowband speech energy Ps for each frequency bin k. nb In some embodiments, the speech energy updating component 504 may change the smoothing factor (α ps ) is applied to narrowband speech energy update: Ps nb (l, k) = α ps Ps nb (l-1, k)+(1-α ps )|X 1 (l, k)|

[0062] The narrowband SNR estimation component 506 is configured to estimate the SNR based on the narrowband noise floor NF nb (l, k) and narrowband speech energy Ps nb (l, k) to estimate the audio signal X 1 The narrowband SNR of (l, k). For example, SNR nb (l, k) can be estimated as: where ε is a small positive number used to avoid division by infinity.

[0063] Figure 5B 1 shows a block diagram of an example wideband SNR detection system 510 according to some embodiments. The wideband SNR detection system 510 is configured to detect an audio signal (X) 1(l, k)) determines the wideband SNR value (SNR wb (l)). More specifically, the broadband SNR detection system 510 can be based on the audio signal X 1 The broadband energy (X1 tot (l) Determine SNR wb In some embodiments, the broadband SNR detection system 510 may be Figure 4 An example of a wideband SNR detector 450 is shown in FIG. Figure 3 , audio signal X 1 (l, k) may be a reference audio signal of the multi-channel audio signal X(l, k).

[0064] The wideband SNR detection system 510 includes a noise floor updating component 512, a speech energy updating component 514, and a wideband SNR estimation component 516. The noise floor updating component 512 is configured to estimate the audio signal X based on the VAD parameters (VAD(l)). 1 The broadband noise floor (NF) of (l, k) wb (l)). For example, refer to Figure 4 , the VAD parameter VAD(l) may be generated by the VAD 410. More specifically, the VAD parameter VAD(l) may indicate the 1 In some embodiments, when the VAD parameter VAD(l) indicates whether speech is present or absent in each frame l of the audio signal X 1 When there is speech in (l, k), the noise floor updating component 512 can avoid updating the broadband noise floor NF wb (l): NF wb (l) = NF wb (l-1) If VAD(l) = 1

[0065] When the VAD parameter VAD(l) indicates that the audio signal X 1 When there is no speech in (l, k) (such as when VAD(l)=0), the noise floor update component 512 can estimate the audio signal X 1 The broadband noise floor NF of the current frame l of (l, k) wb (l). In some embodiments, the noise floor updating component 512 can be based on the estimated broadband noise floor NF wb (l) Is it lower than the broadband energy level X1? tot (l), the upward smoothing factor (α up ) or downward smoothing factor (α dn ) is applied to the broadband noise floor update, where α up >α dn :

[0066] The speech energy updating component 514 is configured to estimate the audio signal X based on the VAD parameter VAD(l) 1 The broadband speech energy (Ps wb In some embodiments, when the VAD parameter VAD(1) ​​indicates that the audio signal X 1 When there is no speech in (l, k), the speech energy updating component 514 can avoid updating the broadband speech energy Ps wb (l): Ps wb (l) = Ps wb (l-1) If VAD(l) = 0

[0067] When the VAD parameter VAD(l) indicates that 1 When speech is present in (l, k) (such as when VAD(l)=1), the speech energy update component 514 can estimate the audio signal X 1 The broadband speech energy Ps of the current frame l of (l, k) wb (l). In some embodiments, the speech energy updating component 514 may add the smoothing factor (α ps ) is applied to narrowband speech energy update: Ps wb (l) = α ps Ps wb (l-1)+(1-α ps )X1 tot (l)

[0068] The wideband SNR estimation component 516 is configured to estimate the SNR based on the wideband noise floor NF wb (l) and broadband speech energy Ps wb (l) to estimate the audio signal X 1 The wideband SNR of (l, k). For example, SNR wb (l) can be estimated as: where ε is a small positive number used to avoid division by infinity.

[0069] Figure 6 1 shows a block diagram of an example adaptive beamforming system 600 according to some embodiments. The adaptive beamforming system 600 is configured to be based at least in part on a microphone array (such as Figure 1 The microphones 112 and 114 or Figure 2 A plurality of (M) audio signals (X) received by microphones 210 (1)-210 (M)) 1(l, k)-X M (l, k)) and via an auxiliary microphone such as Figure 1 The auxiliary audio signal (X) received by the auxiliary microphone 116 aux (l, k)) to generate an enhanced audio signal (Y(l, k)). In some embodiments, the adaptive beamforming system 600 may be Figure 3 An example of a spatial filter 320 is shown in FIG. Figure 3 , audio signal X 1 (l, k)-X M Each of (l, k) may represent a corresponding channel of the multi-channel audio signal X(l, k).

[0070] The adaptive beamforming system 600 includes a reference microphone replacement component 610, an MVDR beamforming component 620, and an RTF estimation component 630. The reference microphone replacement component 610 is configured to generate an adaptive beamforming signal based on a reference audio signal X(l, k) of the multi-channel audio signal X(l, k). 1 (l, k) and auxiliary audio signal X aux (l, k) to generate an adjusted SNR reference audio signal As reference Figure 2 and Figure 3 The reference audio signal X 1 (l, k) represents the audio signal received via the reference microphone of the microphone array, and the reference microphone can be any microphone of the microphone array, which is used as the reference microphone for calculating the RTF vector The adjusted SNR reference audio signal Combined with the remaining audio signal received from the microphone array to produce an adjusted SNR multi-channel audio signal in:

[0071] In some aspects, the reference microphone replacement component 610 can detect the low SNR based on the broadband low SNR detection flag (D wb (l)) The reference audio signal X 1 (l, k) is selectively replaced by the auxiliary audio signal X aux (l, k), to generate an adjusted SNR reference audio signal As reference Figure 3 and Figure 4 As mentioned above, the detection mark D wb (l) indicates the reference audio signal X 1 Whether the wideband SNR of (l, k) is lower than the threshold wideband SNR level. Figure 3 , detection mark D wb(l) may be included in, or otherwise indicated by, the low SNR signal 302. Aspects of the present disclosure recognize that the auxiliary audio signal X aux (l, k) may have a larger value than the reference audio signal X 1 (l, k) has a higher SNR, but can span a narrower frequency range (such as 0≤k≤K aux ). For example, microphones in a microphone array are generally more capable than bone conduction microphones or internal microphones (K>K aux ) detects higher audio frequencies (such as up to k=K).

[0072] Therefore, only when the detection flag D wb (l) indicates that a low SNR condition is detected (D wb (l) = 1), the reference microphone replacement component 610 can use the auxiliary audio signal X aux (l, k) replace or substitute the reference audio signal X 1 In other words, if the detection flag D wb (l) indicates that no low SNR condition is detected (D wb (l)=0), the reference microphone replacement component 610 can convert the reference audio signal X 1 (l, k) output is the adjusted SNR reference audio signal

[0073] In some embodiments, when the detection marker D wb (l) indicates that a low SNR condition is detected (D wb (l) = 1), the reference microphone replacement component 610 can use the auxiliary audio signal X aux (l, k) only replaces or substitutes the reference audio signal X 1 For example, the reference microphone replacement component 610 may replace the auxiliary audio signal X with the auxiliary audio signal X only for a narrow frequency range that can be detected by the auxiliary microphone. aux (l, k) replaces the reference audio signal X 1 (l, k):

[0074] exist Figure 6 In the example of FIG. 1 , the reference microphone replacement component 610 is shown as receiving a single auxiliary audio signal X from a single auxiliary microphone. aux (l, k). However, in some other embodiments, the reference microphone replacement component 610 can receive multiple auxiliary audio signals from multiple auxiliary microphones (such as from a bone conduction microphone and an internal microphone). In such an embodiment, when a low SNR condition (Dwb (l) = 1), the reference microphone replacement component 610 can convert the reference audio signal X 1 (l, k) are replaced by a plurality of auxiliary audio signals. For example, for frequencies below 800 Hz, the reference microphone replacement component 610 can replace the reference audio signal X with an auxiliary audio signal received from a bone conduction microphone. 1 (l, k), and for frequencies between 800 Hz and 1.5 kHz, the reference microphone replacement component 610 can replace the reference audio signal X with the auxiliary audio signal received from the internal microphone 1 (l, k).

[0075] MVDR beamforming component 620 applies MVDR beamforming filter w MVDR (l, k) is applied to the adjusted SNR multi-channel audio signal To generate an enhanced audio signal Y(l, k) (such as according to Equation 2). In some embodiments, the MVDR beamforming component 620 may be Figure 2 For example, the MVDR beamforming component 620 may be based on the audio signal The covariance of the noise in NN (l, k) and the covariance Φ of the speech SS (l, k) Determine the MVDR beamforming filter w MVDR (l, k) filter coefficients (such as according to Equation 4). In some embodiments, the MVDR beamforming component 620 can be based on the audio signal The vector of the associated RTF 602 determines the MVDR beamforming filter w MVDR (l, k) (such as according to Equation 6).

[0076] The RTF estimation component 630 may be configured to update the RTF 602 so that the MVDR beamforming filter w MVDR The beam direction of (l, k) is adapted to the direction of the target speech. For example, the RTF estimation component 630 can be based at least in part on the audio signal The covariance Φ of the speech in SS (l, k) to estimate the RTF vector ) (such as according to Equation 5). Figure 2 As mentioned above, when the audio signal When there is speech in , the speech covariance Φ can be estimated SS (l, k), and when the audio signal When there is no speech in the noise covariance Φ, the noise covariance Φ can be estimated NNIn some implementations, the RTF estimation component 630 may use VAD to determine the audio signal Whether speech is present or absent in Figure 4 However, when the audio signal When the SNR is too low, the RTF estimation component 630 may not be able to accurately estimate the speech covariance Φ SS (l, k).

[0077] In some aspects, the RTF estimation component 630 can be based on the narrowband low SNR detection flag (D nb (l, k)) Selectively update RTF 602. Figure 3 and Figure 4 As mentioned above, the detection mark D nb (l, k) indicates the reference audio signal X 1 Whether the narrowband SNR of (l, k) is lower than the threshold narrowband SNR level. Figure 3 , detection mark D nb (l, k) may be included in, or otherwise indicated by, the low SNR signal 302. In some implementations, only when the audio signal The RTF estimation component 630 can update the RTF 602 only when the SNR of the detection flag D is high enough. nb (l, k) indicates that no low SNR condition is detected (D nb (l, k) = 0), the RTF estimation component 630 may estimate the RTF vector Provided to the MVDR beamforming component 620 (as a vector of RTF 602).

[0078] In contrast, when the audio signal When the SNR of is too low, the RTF estimation component 630 can pause or otherwise avoid updating the RTF 602. In some embodiments, when the detection flag D nb (l, k) indicates that a low SNR condition (D nb (l, k) = 1), the RTF estimation component 630 can be a predetermined RTF vector Provided to the MVDR beamforming component 620 (as a vector of RTF 602). Real-time estimated RTF vector Different, pre-defined RTF vector Does not depend on the current audio signal For example, the predefined RTF vector may be stored by RTF estimation component 630 (such as in RTF storage 632). can be any RTF vector known to result in a relatively accurate beam direction. In some aspects, the predetermined RTF vector may be the last RTF vector estimated by the RTF estimation component 630 before pausing updates to the RTF 602.

[0079] In some other aspects (such as when the estimated RTF vector Not yet available), a predefined RTF vector can be configured based on the geometry of the user's head or microphone array For example, reference Figure 1 , the headset 110 is designed to be worn in substantially the same position on any user's head. As such, aspects of the present disclosure recognize that the relative positioning of the microphones 112 and 114 with respect to the mouth of the user 120 may vary little, if at all, over time and may be substantially the same for different users. Thus, in some embodiments, the predetermined RTF vector may be estimated by testing the headset 110 on multiple users (including males and females) with different head shapes and sizes. In some other embodiments, a predetermined RTF vector may be estimated and stored for a particular user of the headset 110 via an initial calibration process.

[0080] As a result, when the audio signal When the SNR is high enough (such as when the detection mark D nb When (l, k) indicates a low SNR condition is detected), the MVDR beamforming component 620 may be based on the estimated RTF vector To determine the MVDR beamforming filter w MVDR (l, k). In contrast, when the audio signal When the SNR is too low (such as when the detection mark D nb (l, k) indicates that no low SNR condition is detected), the MVDR beamforming component 620 may be based on a predetermined RTF vector Determine the MVDR beamforming filter w MVDR (l, k). Accordingly, the MVDR beamforming filter w MVDR (l, k) can be expressed as the detection mark D nb Function of (l, k):

[0081] As reference Figure 2 The audio signal can be based on The covariance Φ of the speech in SS (l, k) to estimate the RTF vector Therefore, using Equation 4, the MVDR beamforming filter w MVDR (l, k) can be rewritten as the speech covariance Φ SS (l, k) and the one-hot vector (u(l, k)) representing the reference microphone channel:

[0082] Figure 7 Another block diagram of an example speech enhancement system 700 according to some embodiments is shown. The speech enhancement system 700 is configured to enhance the speech component of a multi-channel audio signal based at least in part on an auxiliary audio signal. The multi-channel audio signal may be received via a microphone array, and the auxiliary audio signal may be received via an auxiliary microphone separate from the microphone array. In some embodiments, the microphone array may be Figure 1 The microphones 112 and 114 or Figure 2 An example of microphones 210(1)-210(M) and the auxiliary microphone may be Figure 1 An example of an auxiliary microphone 116.

[0083] The speech enhancement system 700 includes a device interface 710, a processing system 720, and a memory 730. The device interface 710 is configured to communicate with an audio receiver (such as Figure 1 The device interface 710 may include a microphone array interface (I / F) 712 configured to communicate with a microphone array and an auxiliary microphone interface (I / F) 714 configured to communicate with an auxiliary microphone. The microphone array interface 712 may receive multiple audio signals via multiple microphones of the microphone array, respectively, wherein each of the multiple audio signals represents a corresponding channel of the multi-channel audio signal. The auxiliary microphone interface 714 may receive an auxiliary audio signal via an auxiliary microphone.

[0084] The memory 730 may include an audio data storage 732 configured to store frames of the multi-channel audio signal and the auxiliary audio signal and any intermediate signals that may be generated by the speech enhancement system 700 as a result of speech enhancement. The memory 730 may also include a non-transitory computer-readable medium (including one or more non-volatile memory elements such as EPROM, EEPROM, flash memory, or hard drive, among other examples) that may store at least the following software (SW) modules: An SNR detection SW module 734 for detecting a wideband SNR of a reference audio signal of a plurality of audio signals; a reference microphone replacement SW module 736 for selectively replacing at least part of the reference audio signal with the auxiliary audio signal based on the wideband SNR, such that as a result of said replacement the multi-channel audio signal comprises the auxiliary audio signal in place of at least part of the reference audio signal; and A speech enhancement SW module 738 is used to enhance the speech component of the multi-channel audio signal based on the MVDR beamforming filter. Each software module includes instructions that, when executed by the processing system 720, cause the speech enhancement system 700 to perform a corresponding function.

[0085] The processing system 720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the speech enhancement system 700 (such as in the memory 730). For example, the processing system 720 may execute the SNR detection SW module 734 to detect the wideband SNR of the reference audio signal of the multiple audio signals. The processing system 720 may also execute the reference microphone replacement SW module 736 to selectively replace at least a portion of the reference audio signal with the auxiliary audio signal based on the wideband SNR, so that as a result of the replacement, the multi-channel audio signal includes the auxiliary audio signal to replace at least a portion of the reference audio signal. In addition, the processing system 720 may execute the speech enhancement SW module 738 to enhance the speech component of the multi-channel audio signal based on the MVDR beamforming filter.

[0086] Figure 8 An illustrative flow chart depicting example operations 800 for speech enhancement according to some implementations is shown. In some implementations, the example operations 800 may be performed by a speech enhancement system, such as Figure 3 Speech Enhancement System 300 or Figure 7 Speech enhancement system 700.

[0087] The speech enhancement system receives multiple audio signals via multiple microphones of the microphone array, respectively, wherein each of the multiple audio signals represents a corresponding channel of the multi-channel audio signal (810). The speech enhancement system also receives an auxiliary audio signal (820) via an auxiliary microphone separate from the microphone array. In some aspects, the microphone array can be disposed on an outer surface of a shell worn by a user, and the auxiliary microphone can be disposed on an inner surface of the shell closer to the user than the outer surface. In some embodiments, the auxiliary microphone can be a bone conduction microphone. In some other embodiments, the auxiliary microphone can be a feedback microphone associated with the ANC system.

[0088] The speech enhancement system detects a wideband signal-to-noise ratio (SNR) of a reference audio signal of the plurality of audio signals (830). In some implementations, the wideband SNR may be detected based on a noise floor of the reference audio signal. The speech enhancement system selectively replaces at least a portion of the reference audio signal with an auxiliary audio signal based on the wideband SNR, such that as a result of the replacement, the multi-channel audio signal includes the auxiliary audio signal in place of at least a portion of the reference audio signal (840). The speech enhancement system further enhances a speech component of the multi-channel audio signal based on an MVDR beamforming filter (850).

[0089] In some aspects, the speech enhancement system can determine whether the wideband SNR is below a threshold level, and replace at least a portion of the reference audio signal with the auxiliary audio signal in response to determining that the wideband SNR is below the threshold level. In some embodiments, each of the multiple audio signals can be associated with a first range of frequencies, and the auxiliary audio signal can be associated with a second range of frequencies that is narrower than the first range. In such embodiments, the portion of the reference audio signal that is replaced with the auxiliary audio signal can include any frequency components of the reference audio signal that overlap with the second range of frequencies.

[0090] In some aspects, the speech enhancement system may determine a plurality of RTFs based on the multi-channel audio signal, determine an MVDR beamforming filter based at least in part on the plurality of RTFs, detect a narrowband SNR of a reference audio signal, determine whether the narrowband SNR is below a threshold level, and selectively update the plurality of RTFs based on whether the narrowband SNR is below the threshold level. In some embodiments, the speech enhancement system may refrain from updating the plurality of RTFs in response to determining that the narrowband SNR is below the threshold level. In some other embodiments, the speech enhancement system may dynamically update the plurality of RTFs in response to determining that the narrowband SNR is not below the threshold level.

[0091] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0092] In addition, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present disclosure.

[0093] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or in any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor.

[0094] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for speech enhancement, comprising: receiving a plurality of audio signals via a plurality of microphones of the microphone array, respectively, each of the plurality of audio signals representing a corresponding channel of the multi-channel audio signal; receiving an auxiliary audio signal via an auxiliary microphone separate from the microphone array; detecting a wideband signal-to-noise ratio (SNR) of a reference audio signal of the plurality of audio signals; selectively replacing at least a portion of the reference audio signal with the auxiliary audio signal based on the wideband SNR, such that as a result of the replacing, the multi-channel audio signal includes the auxiliary audio signal in place of the at least a portion of the reference audio signal; as well as Based on a minimum variance distortionless response (MVDR) beamforming filter, a speech component of the multi-channel audio signal is enhanced.

2. The method of claim 1, wherein the microphone array is disposed on an outer surface of a housing worn by a user, and the auxiliary microphone is disposed on an inner surface of the housing closer to the user than the outer surface. The method of claim 1 , wherein the auxiliary microphone comprises a bone conduction microphone.

4. The method of claim 1, wherein the auxiliary microphone comprises a feedback microphone associated with an active noise cancellation (ANC) system. The method of claim 1 , wherein the wideband SNR is detected based on a noise floor of the reference audio signal.

6. The method of claim 1 , wherein the selective replacement of at least a portion of the reference audio signal comprises: determining whether the wideband SNR is below a threshold level; as well as At least a portion of the reference audio signal is replaced with the auxiliary audio signal in response to determining that the wideband SNR is below the threshold level.

7. The method of claim 1, wherein each of the plurality of audio signals is associated with a first range of frequencies and the auxiliary audio signal is associated with a second range of frequencies that is narrower than the first range.

8. The method of claim 7, wherein the portion of the reference audio signal that is replaced with the auxiliary audio signal includes any frequency components of the reference audio signal that overlap with the second range of frequencies.

9. The method according to claim 1, further comprising: determining a plurality of relative transfer functions (RTFs) based on the multi-channel audio signal; determining the MVDR beamforming filter based at least in part on the plurality of RTFs; Detecting a narrowband SNR of the reference audio signal; determining whether the narrowband SNR is below a threshold level; as well as The plurality of RTFs are selectively updated based on whether the narrowband SNR is below the threshold level.

10. The method of claim 9, wherein the selective updating of the plurality of RTFs comprises: The plurality of RTFs are dynamically updated in response to a determination that the narrowband SNR is not below the threshold level.

11. The method of claim 9, wherein the selective updating of the plurality of RTFs comprises: Updating the plurality of RTFs is refrained in response to determining that the narrowband SNR is below the threshold level.

12. A speech enhancement system, comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the speech enhancement system to: receiving a plurality of audio signals via a plurality of microphones of the microphone array, respectively, each of the plurality of audio signals representing a corresponding channel of the multi-channel audio signal; receiving an auxiliary audio signal via an auxiliary microphone separate from the microphone array; detecting a wideband signal-to-noise ratio (SNR) of a reference audio signal of the plurality of audio signals; selectively replacing at least a portion of the reference audio signal with the auxiliary audio signal based on the wideband SNR, such that as a result of the replacing, the multi-channel audio signal includes the auxiliary audio signal in place of the at least a portion of the reference audio signal; as well as Based on a minimum variance distortionless response (MVDR) beamforming filter, a speech component of the multi-channel audio signal is enhanced.

13. The speech enhancement system of claim 12, wherein the microphone array is disposed on an outer surface of a housing worn by a user, and the auxiliary microphone is disposed on an inner surface of the housing closer to the user than the outer surface.

14. The speech enhancement system of claim 12, wherein the auxiliary microphone comprises a bone conduction microphone or a feedback microphone associated with an active noise cancellation (ANC) system.

15. The speech enhancement system of claim 12, wherein the wideband SNR is detected based on a noise floor of the reference audio signal.

16. The speech enhancement system of claim 12, wherein the selective replacement of at least a portion of the reference audio signal comprises: determining whether the wideband SNR is below a threshold level; as well as At least a portion of the reference audio signal is replaced with the auxiliary audio signal in response to determining that the wideband SNR is below the threshold level.

17. The speech enhancement system of claim 12, wherein each of the plurality of audio signals is associated with a first range of frequencies and the auxiliary audio signal is associated with a second range of frequencies narrower than the first range, the portion of the reference audio signal that is replaced with the auxiliary audio signal including any frequency components of the reference audio signal that overlap with the second range of frequencies.

18. The speech enhancement system of claim 12, wherein execution of the instructions further causes the speech enhancement system to: determining a plurality of relative transfer functions (RTFs) based on the multi-channel audio signal; determining the MVDR beamforming filter based at least in part on the plurality of RTFs; Detecting a narrowband SNR of the reference audio signal; determining whether the narrowband SNR is below a threshold level; as well as The plurality of RTFs are selectively updated based on whether the narrowband SNR is below the threshold level.

19. The speech enhancement system of claim 18, wherein the selective updating of the plurality of RTFs comprises: The plurality of RTFs are dynamically updated in response to a determination that the narrowband SNR is not below the threshold level.

20. The speech enhancement system of claim 18, wherein the selective updating of the plurality of RTFs comprises: Updating the plurality of RTFs is refrained in response to determining that the narrowband SNR is below the threshold level.

Images