Audio system and method for voice activity detection

Abstract

Audio systems, methods, and processor instructions are provided herein that detect voice activity of a user and provide an output voice signal. The systems, methods, and instructions receive a plurality of microphone signals and combine the plurality of microphone signals according to a first combination and a second combination. The first combination produces a primary signal having an enhanced response in a direction of a mouth of the user, and the second combination produces a reference signal having a reduced response in the direction of the mouth of the user. The primary signal and the reference signal are added and subtracted to produce a sum signal and a difference signal, respectively. The sum signal is compared to the difference signal and an output voice signal is provided based on the comparison.

Description

Background Technology

[0001] Various audio devices, such as headphones and earphones, are used in numerous environments for a variety of purposes, including entertainment (such as gaming or listening to music), production (such as telephone calls), and professional (such as aviation communications or studio monitoring). Different environments and purposes may have different requirements for fidelity, noise isolation, noise reduction, and voice pickup. Various echo and noise cancellation and reduction systems and methods, as well as other processing systems and methods, may be included to improve accurate communication when providing users with voice or speech output signals.

[0002] Some of these systems and methods exhibit improved performance when they have a reliable indication that the user of the device is actively speaking. For example, upon reliably determining that the user is speaking, certain systems and methods can modify various processing parameters, such as filter coefficients, adaptive rates, reference signal selection, etc. This enhanced performance allows the user's speech to be more clearly separated or isolated from other noise in the output audio signal, further enabling enhanced applications such as voice communication and speech recognition, including speech recognition for communication, such as voice-to-text applications for Short Message Service (SMS) or Virtual Personal Assistant (VPA) applications.

[0003] Therefore, there is a need for reliable detection of a user’s speech, and this application relates to reliable detection of a user’s speech, which is generally referred to herein as voice activity detection (VAD). Summary of the Invention

[0004] Aspects and examples relate to audio systems and methods that pick up a user's speech from one or more microphone signals and reduce other acoustic components (such as background noise and other speakers) to enhance the user's speech component rather than the other acoustic components. More specifically, aspects and examples relate to methods and systems for reliably detecting when a user is speaking (i.e., speech activity detection).

[0005] According to one aspect, a method for detecting a user's voice activity is provided, and the method includes: receiving a plurality of microphone signals; combining the plurality of microphone signals according to a first combination to generate a main signal having an enhanced response in the direction of the user's mouth; combining the plurality of microphone signals according to a second combination to generate a reference signal having a reduced response in the direction of the user's mouth; adding the main signal and the reference signal to generate a summed signal; subtracting one of the main signal or the reference signal from the other to generate a difference signal; comparing the summed signal with the difference signal; and providing an output speech signal based on the comparison.

[0006] In various examples, the first combination can be a minimum variance distortion-free response (MVDR) combination. The second combination can be a delay and subtraction combination.

[0007] According to some examples, comparing the summed signal with the difference signal includes determining at least one of the energy, amplitude, or envelope of each of the summed signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summed signal and the difference signal. This comparison may also include comparing at least one of the ratios or differences with a threshold, or multiplying at least one of the energy, amplitude, or envelope by a factor, and comparing the multiplied energy, amplitude, or envelope with other energies, amplitudes, or envelopes.

[0008] In various examples, comparing the summed signal with the difference signal includes comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, which is different from the first frequency band. In some examples, the first frequency band may include frequencies in the range of 200Hz-400Hz, and the second frequency band may include frequencies in the range of 500Hz-700Hz.

[0009] Some examples may include processing a speech signal with an adaptive filter and modifying the adaptive filter based on the comparison. Modifying the adaptive filter may include changing the coefficients of the adaptive filter, changing the adaptive rate, changing the step size, freezing the adaptive filter, or disabling the adaptive filter.

[0010] According to another aspect, an audio system is provided, comprising a plurality of microphones and a controller coupled to the plurality of microphones. The controller is configured to: receive a plurality of microphone signals from the plurality of microphones; combine the plurality of microphone signals according to a first combination to generate a master signal having an enhanced response in the direction of the user's mouth; combine the plurality of microphone signals according to a second combination to generate a reference signal having a reduced response in the direction of the user's mouth; add the master signal and the reference signal to generate a summed signal; subtract one of the master signal or the reference signal from the other to generate a difference signal; compare the summed signal with the difference signal; and provide an output speech signal based on the comparison.

[0011] In some examples, the first combination can be a minimum variance distortion-free response (MVDR) combination, and the second combination can be a delay and subtraction combination.

[0012] In various examples, comparing the summed signal with the difference signal includes determining at least one of the energy, amplitude, or envelope of each of the summed signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summed signal and the difference signal.

[0013] In various examples, comparing the summed signal with the difference signal includes comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, which is different from the first frequency band. For example, in some examples, the first frequency band may include frequencies in the range of 200Hz-400Hz, and the second frequency band may include frequencies in the range of 500Hz-700Hz.

[0014] In some examples, providing the speech signal based on the comparison may include processing the speech signal with an adaptive filter and changing the adaptive filter based on the comparison. Changing the adaptive filter may include changing the coefficients of the adaptive filter, changing the adaptive rate, changing the step size, freezing the adaptive filter, or disabling the adaptive filter.

[0015] According to another aspect, a non-transitory computer-readable medium having instructions encoded thereon is provided, which, when executed by a suitable processor (or processors), cause the processor to perform a method comprising: receiving a plurality of microphone signals; combining the plurality of microphone signals according to a first combination to generate a main signal having an enhanced response in the direction of the user's mouth; combining the plurality of microphone signals according to a second combination to generate a reference signal having a reduced response in the direction of the user's mouth; adding the main signal and the reference signal to generate a summed signal; subtracting one of the main signal or the reference signal from the other to generate a difference signal; comparing the summed signal with the difference signal; and providing an output speech signal based on the comparison.

[0016] In various examples, the first combination can be a minimum variance distortion-free response (MVDR) combination. The second combination can be a delay and subtraction combination.

[0017] According to some examples, comparing the summed signal with the difference signal includes determining at least one of the energy, amplitude, or envelope of each of the summed signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summed signal and the difference signal. This comparison may also include comparing at least one of the ratios or differences with a threshold, or multiplying at least one of the energy, amplitude, or envelope by a factor, and comparing the multiplied energy, amplitude, or envelope with other energies, amplitudes, or envelopes.

[0018] In various examples, comparing the summed signal with the difference signal includes comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, which is different from the first frequency band. In some examples, the first frequency band may include frequencies in the range of 200Hz-400Hz, and the second frequency band may include frequencies in the range of 500Hz-700Hz.

[0019] Some examples may include processing a speech signal with an adaptive filter and modifying the adaptive filter based on the comparison. Modifying the adaptive filter may include changing the coefficients of the adaptive filter, changing the adaptive rate, changing the step size, freezing the adaptive filter, or disabling the adaptive filter.

[0020] Other aspects, examples, and advantages of these exemplary aspects and examples are discussed in detail below. The examples disclosed herein may be combined with other examples in any manner consistent with at least one of the principles disclosed herein, and references to “example,” “some examples,” “alternative examples,” “various examples,” “an example,” etc., are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or property described may be included in at least one example. The appearance of such terms in this document does not necessarily refer to the same example. Attached Figure Description

[0021] The following discussion of at least one example, with reference to the accompanying drawings, is not intended to be drawn to scale. The drawings are included to provide illustration and further understanding of the various aspects and examples, and are incorporated in and form part of this specification, but are not intended to be a definition of limitation of the invention. In the drawings, the same or nearly identical parts shown in the various figures may be represented by similar numbers. For clarity, not every part is labeled in every figure. In the drawings:

[0022] Figure 1 A perspective view of a pair of exemplary headphones;

[0023] Figure 2 For which it can be used Figure 1 A schematic diagram of the environment of an exemplary pair of headphones;

[0024] Figure 3 A schematic diagram of an exemplary noise reduction system for enhancing a user's speech signal in other acoustic signals;

[0025] Figure 4 A schematic diagram of an exemplary system for detecting a user's voice activity;

[0026] Figure 5 This is a schematic diagram of another exemplary system for detecting a user's voice activity; and

[0027] Figure 6 This is a flowchart of an exemplary voice activity detection method. Detailed Implementation

[0028] Various aspects of this disclosure relate to audio systems and methods that support the pickup of speech signals from users (e.g., wearers) of headsets, headphones, etc., by reliably detecting user speech activity (e.g., detecting when a user is speaking). Conventional speech activity detection (VAD) systems and methods may receive or construct a main signal configured or arranged to include user speech components, and receive or construct a reference signal configured or arranged to exclude (or reduce the inclusion of) user speech components. The signal envelope, amplitude, or energy of the main signal is compared with the signal envelope, amplitude, or energy of the reference signal, and if the main signal exceeds a threshold relative to the reference signal, it is determined that the user is speaking. Such systems and methods typically output a binary flag (e.g., VAD = 0, 1) to indicate whether the user is speaking. This flag may be advantageously applied to other parts of the audio system to freeze the adaptation of adaptive filters in noise cancellation or reduction systems and / or echo cancellers. The application of the VAD indication may cover a number of other actions or effects beyond the scope of this disclosure but which will be apparent to those skilled in the art.

[0029] Conventional VAD systems and methods that conform to the above descriptions may encounter performance degradation issues when the audio system is near boundary conditions (e.g., acoustically reflective environments, such as nearby walls and / or the user's arm, hand, etc., placed near headphones, earphones, etc.). Essentially, acoustic reflections of user speech from boundary conditions can enter the reference signal, thus reducing the differential signal energy between the main signal (intended to include the user's speech) and the reference signal (intended not to include the user's speech). The aspects and examples described herein accommodate this phenomenon and enhance the reliability of speech activity detection when the user is near or creates boundary conditions (e.g., relative to nearby acoustically reflective objects or surfaces).

[0030] Obtaining user speech signals with reduced noise and / or echo components can enhance voice-based features or functions that can be provided as part of an audio system or other associated equipment, such as communication systems (cellular, radio, aviation), entertainment systems (gaming), speech recognition applications (speech-to-text, virtual personal assistants), and other systems and applications that process audio (especially speech or voice). The examples disclosed herein can be coupled to or connected to other systems via wired or wireless means, or can operate independently of other systems or equipment.

[0031] Headphones, earphones, headsets and other various personal audio system form factors (e.g., in-ear transducers, earbuds, neckband or shoulder-worn devices and other headband devices with integrated audio, glasses, etc.) conform to various aspects and examples in this document.

[0032] Generally speaking, acoustic reflections from nearby environmental boundaries (e.g., surfaces and objects) can cause a significant reduction in the performance of conventional VAD in unilateral (e.g., left or right) audio systems compared to binaural audio systems (left and right), because the additional signal characteristics between the left and right sides may not be available in unilateral systems and methods. Therefore, the aspects and examples disclosed herein may be more suitable for unilateral audio systems and methods. Nevertheless, the aspects and examples described can also be applied to binaural systems and methods.

[0033] The examples disclosed herein may be combined with other examples in any way consistent with at least one of the principles disclosed herein, and references to “example,” “some examples,” “alternative examples,” “various examples,” “an example,” etc., are not necessarily mutually exclusive, but are intended to indicate that the particular feature, structure, or property described may be included in at least one example. The appearance of such terms in this document does not necessarily refer to the same example.

[0034] It should be understood that the examples of methods and apparatus discussed herein are not limited to the construction details and component arrangements listed in the following description or shown in the accompanying drawings. These methods and apparatuses can be implemented in other examples and can be operated or performed in various ways. The examples of specific implementations provided herein are for illustrative purposes only and are not intended to be limiting. Furthermore, the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising,” “including,” “having,” “containing,” “involving,” and variations thereof herein is intended to cover the items listed thereafter and their equivalents, as well as additional items. References to “or” are to be understood as inclusive, such that any term described using “or” can indicate any one, more than one, or all of the terms mentioned. Any references to front and back, right and left, top and bottom, upper and lower, and vertical and horizontal are for ease of description and not for limiting the system and methods or their components to any one location or spatial orientation.

[0035] Figure 1An example of an earplug 100 is shown, comprising an earplug end 110, an acoustic transducer (speaker, internal and therefore not shown) for generating an acoustic output from, for example, an audio signal, and one or more microphones 120. While the exemplary earplug 100 is shown for the right ear, examples for the left ear may also be provided, for example, in a symmetrical or mirror-image manner, and / or various examples may include a pair of left and right earplugs. Generally, the earplug end 110 includes an acoustic channel and an end having, for example, an "umbrella" feature configured to provide a level of acoustic seal near the ear canal of the user (e.g., the wearer) of the earplug 100. The earplug end also includes retaining and stabilizing features, such as two arms joined at the distal end, to hold the earplug 100 in the user's ear during use. Other examples may include different support structures to hold one or more earpieces near the user's ear. For example, open-ear audio devices may be incorporated into eyeglasses or other head-mounted devices and / or worn in structures near or around the head, neck, and / or ears.

[0036] Earplug 100 is shown having two microphones 120, a forward-facing microphone 120F and a rearward-facing microphone 120R (collectively referred to as 120). In other examples, more microphones may be included and may be arranged in varying positions. The microphones 120 are positioned in varying positions such that they do not receive the same acoustic signal. It is advantageous to compare combinations of variations in signals from two or more microphones to detect whether a user is speaking, provide a speech signal representing the user's voice, remove or reduce noise and / or echo components from the speech signal, and various other signal processing and / or communication functions and features.

[0037] Although microphones are shown and labeled with reference numerals, in some examples, the visual elements shown in the figures may represent acoustic ports through which acoustic signals enter to ultimately reach microphones, which may be internal and physically invisible from the outside. In the examples, one or more microphones in microphone 120 may be located inside an acoustic port or may be moved a distance away from the acoustic port, and may include acoustic waveguides between the acoustic port and the associated microphones.

[0038] Signals from microphone 120 are combined in various ways to advantageously guide the beam and null values ​​in a manner that maximizes the user's speech in one instance to provide the master signal and minimizes the user's speech in another instance to provide the reference signal. Thus, the reference signal can represent ambient noise and can be provided as a reference to the adaptive filter of the noise reduction subsystem. Such a noise reduction system can modify the master signal to reduce components associated with the reference signal (e.g., noise-dependent signals), and the noise reduction subsystem provides an output signal that approximates the user's speech signal, with reduced noise content.

[0039] In various examples, signals can be advantageously processed in different subbands to enhance the effectiveness of noise reduction or other signal processing. The generation of a signal in which the user's speech component is enhanced while other components are reduced is generally referred to herein as speech pickup, speech selection, speech isolation, speech enhancement, etc. As used herein, the terms "speech," "voice," "conversation," and their variants are used interchangeably regardless of whether the speech involves the use of the vocal cords.

[0040] Figure 2 An exemplary environment 200 is illustrated, in which a user 210 (shown as a top view of the user's head) may wear an audio device (such as earplugs 100) near an acoustically reflective surface 220 (such as a wall). For certain acoustic frequencies, particularly specific frequencies where the distance d (230) between the earplugs 100 and the reflective surface 220 is less than a quarter wavelength, the indirect acoustic energy reflected from the acoustically reflective surface 220 may become substantially in phase with the direct acoustic energy reaching the microphone 120. Therefore, such signal processing may exhibit degraded performance when various signal processing methods for one or more microphone signals or combinations of microphone signals depend on the directionality of various components in the microphone signals. For example, speech activity detectors, noise reduction systems, echo reduction systems, and especially those systems that depend on combinations of microphone signals to enhance or reduce acoustic signals from certain directions (e.g., beamformers and null-forming systems, or typically array processing) may exhibit degraded performance, such as when signal content intended to be excluded by such combinations is instead included because it is reflected by the reflective surface 220. In various examples, the acoustic reflective surface (such as reflective surface 220) can be a wall, corner, half-wall, furniture or other object, headrest, or the user's hand (such as when making a gesture, reaching for earplug 100, or putting a hand behind the head).

[0041] Figure 3 A block diagram of an exemplary noise reduction system 300 for processing microphone signals to produce an output signal that includes a user speech component enhanced relative to background noise and other speakers. A set of multiple microphones 302 (such as...) Figures 1 to 2 The microphone 120 converts acoustic energy into an electronic signal 304 and provides the signal 304 to each of the two array processors 306, 308. The signal 304 may be in analog form. Alternatively, one or more analog-to-digital converters (ADCs) (not shown) may first convert the microphone output so that the signal 304 may be in digital form. The array processors 306, 308 apply array processing techniques, such as phased arrays, delay-addition techniques, and may utilize minimum variance distortion-free response (MVDR) and linearly constrained minimum variance (LCMV) techniques to adjust the responsiveness of the group of microphones 302 to enhance or reject acoustic signals from various directions.

[0042] Beamforming enhances acoustic signals from a specific direction or range of directions, while null formation reduces or rejects acoustic signals from a specific direction or range of directions. The first array processor 306 is a beamformer used to maximize the acoustic response of the group of microphones 302 in the direction of the user's mouth (e.g., pointing in front of and below, for example, earplug 100) and provide a master signal 310. Due to the beamforming array processor 306, the master signal 310 includes a higher level of user speech signal energy than any single microphone signal 304. The master signal 310, as the output of the first array processor 306, can be considered equivalent to the output of a directional microphone pointing towards the user's mouth.

[0043] The second array processor 308 directs a zero value toward the user's mouth and provides a reference signal 312. Since the zero value is directed toward the user's mouth, the reference signal 312 includes the minimum (if any) signal energy of the user's speech. Therefore, the reference signal 312 is essentially composed of components caused by background noise and other sound sources that are not the user's speech. For example, the reference signal 312 is a signal related to the acoustic environment other than the user's speech. The reference signal 312, as the output of the second array processor 308, can be considered equivalent to the output of a microphone directed toward the surrounding environment (anywhere except the user's mouth).

[0044] The main signal 310 includes user speech components and noise components (e.g., background, other speakers, etc.), while under normal circumstances, the reference signal 312 essentially includes only noise components. If the reference signal 312 is nearly identical to the noise components of the main signal 310, the noise components of the main signal 310 can be removed by simply subtracting the reference signal 312 from the main signal 310. However, in practice, the reference signal 312 is correlated with and indicates the noise components of the main signal 310, but is not precisely equal to the noise components of the main signal 310, as those skilled in the art will understand. Therefore, adaptive filtering can be used to remove at least some of the noise components from the main signal 310 by using the reference signal 312 as an indicator of the noise components.

[0045] Many adaptive filter methods known in the art are designed to remove components that are correlated with a reference signal. For example, some examples include the Normalized Least Mean Square (NLMS) adaptive filter. The output of the adaptive filter 314 is a speech estimation signal 316, which represents an approximation of the user's speech signal.

[0046] Exemplary adaptive filter 314 may include various types incorporating various adaptive techniques (e.g., NLMS). The operation of an adaptive filter typically involves a digital filter that receives a reference signal associated with unwanted components of the main signal. The digital filter attempts to generate an estimate of the unwanted components in the main signal from the reference signal. By definition, the unwanted components of the main signal are noise components. The estimate of the noise components by the digital filter is a noise estimate. If the digital filter generates a good noise estimate, the noise component can be effectively removed from the main signal by simply subtracting the noise estimate. On the other hand, if the digital filter does not generate a good estimate of the noise component, this subtraction may be ineffective or may degrade the main signal, for example, by increasing noise. Therefore, the adaptive algorithm operates in parallel with the digital filter and adjusts the digital filter, for example, by changing weights or filter coefficients. In some examples, the adaptive algorithm may monitor the main signal when it is known that only noise components exist (i.e., when the user is not speaking) and adjust the digital filter to generate a noise estimate that matches the main signal, in which case the main signal includes only noise components. The adaptive algorithm can know when the user is not speaking by various means. In at least one example, the system enforces a pause or silence period after triggering speech enhancement. For example, a user might need to press a button or speak a wake-up command, followed by a pause until the system indicates to the user that they are ready. During the desired pause, an adaptive algorithm monitors the main signal, which does not include any user speech, and adapts the filter to background noise. Then, when the user speaks, the digital filter generates a good noise estimate, which is subtracted from the main signal to generate a speech estimate, such as speech estimate signal 316.

[0047] Additionally, and according to the examples in this article, the voice activity detectors 400 and 500 (VAD) are operable to detect when a user is speaking or when they are not speaking. Figure 4 and Figure 5 Each illustrates the operation of an exemplary speech activity detection algorithm. Figure 4 In the example, two microphones (120) are used, but in other examples, an additional microphone may be used. Similar to... Figure 3 The noise reduction system 300, VAD 400 combines microphone signals 404 according to a first combination 406 to generate a main signal 410, and combines microphone signals according to a second combination 408 to generate a reference signal 412. In some examples, the main signal 410 may be the same signal as the main signal 310, but is not required. Similarly, in some examples, the reference signal 412 may be the same signal as the reference signal 312, but is not required.

[0048] The first combination 406 may be an array process that combines the microphone signals 404 to have an enhanced response in the direction of the user's mouth, thereby generating a main signal 410 with enhanced speech components when the user speaks. According to some examples, the first combination 406 may be an MVDR beamformer. The main signal 410, as the output of the first combination 406, can be considered equivalent to the output of a directional microphone pointed towards the user's mouth.

[0049] The second combination 408 may be an array process that combines the microphone signals 404 to have a reduced response in the direction of the user's mouth, thereby generating a reference signal 412 with reduced speech components (and thus with enhanced noise components representing the surrounding environment). In some examples, the second combination 408 may be a zero-value generator with a zero-point (or low) response in the direction of the user's mouth. The reference signal 412, as the output of the second combination 408, can be considered equivalent to the output of a microphone pointed towards the surrounding environment (anywhere except the user's mouth).

[0050] According to at least one example, the second combination 408 can be a combination of the delay and subtraction of the microphone signal 404. (See reference) Figure 1 and Figure 2 When the earbud 100 is worn correctly, the front microphone 120F is closer to the user's mouth than the rear microphone 120R. Therefore, the user's voice reaches the front microphone 120F first, and then the rear microphone 120R. Thus, delaying the signal from the front microphone 120F by an appropriate amount of time (to align the timing of the two microphone signals) and subtracting the other from one microphone signal cancels out the user's speech component. Therefore, in this example, the reference signal 412 has a reduced user speech component.

[0051] Continue to refer to Figure 4In the VAD 400, comparator 414 compares a main signal 410 with a reference signal 412. When the user is not speaking, the main signal 410 and the reference signal 412 may have some relationship with each other, such as their relative energies being substantially constant. However, if the user begins to speak, the energy in the main signal 410 may increase significantly (because it includes the user's speech), while the energy in the reference signal 412 may not increase (because it rejects the user's speech). In a sense, the reference signal 412 may indicate the acoustic environment (e.g., its noise level) from which the comparator 414 can "predict" the baseline signal level in the main signal, and if the main signal 414 exceeds the baseline level, it may be because the user is speaking. Therefore, the comparator 414 can determine whether the user is speaking and provide an output 416 indicating whether speech activity is detected (or not detected). According to various examples, the output 416 may have two states (e.g., logic one or zero) to indicate whether the user is speaking. Other examples may provide various forms of output 416.

[0052] According to various examples, comparator 414 can compare any or more of the energy, amplitude, envelope, or other properties of the signals being compared. Furthermore, comparator 414 can compare signals to each other and / or compare a threshold with any of the signals and / or with any of the ratios or differences between the signals (e.g., the ratio or difference between the energy, amplitude, envelope, etc. of the signals). In various examples, comparator 414 may include signal smoothing, time averaging, or low-pass filtering. In various examples, comparator 414 can perform comparisons within a limited frequency band or sub-band.

[0053] In some examples, it might be desirable for comparator 414 to acquire a ratio of signal energy (or amplitude, envelope, etc.) and compare that ratio to a threshold. Instead of strictly calculating the ratio (which can be computationally resource-intensive), some examples could equivalently adjust a signal attribute by multiplying one of its attributes by a factor and then comparing the adjusted attribute to a comparable attribute of another signal. For example, in some examples, when the signal energy of the primary signal 410 exceeds the energy of the reference signal 412 by a certain amount (or vice versa), say 20%, this can be determined by comparator 414 outputting VAD = 1 (speech detected). In some examples, comparator 414 could determine the signal energy, calculate the ratio of the signal energy, and compare that ratio to a threshold of 1.2 (e.g., representing a 20% increase). However, in some examples, comparator 414 could equivalently multiply one of the signal energies by 1.2 and compare the result directly to another signal energy. For example, multiplication may be less computationally expensive than calculating a ratio between two signal energies.

[0054] The ability to detect speech activity is likely a core control element in various audio systems, and especially in audio systems that include speech pickup and other processing to provide the outgoing user speech signal. For example, an audio system may include one or more subsystems that perform adaptive processing when the user is not speaking, but need to freeze the adaptation when the user begins to speak (e.g., Figure 3 (Noise reduction system 300). Various subsystems may modify their operation in different ways depending on whether the user is speaking and / or may terminate their operation when the user is speaking. For example, in some examples, when the user is not speaking, the outgoing user voice signal may be paused, such as by operating in half-duplex mode to save energy and / or bandwidth. VAD lets the system know to resume transmission. For these and other reasons, effective voice activity detection is essential. Specifically, if VAD fails, the user's voice component may be treated as noise, and adaptive processing may operate unfavorably to remove it.

[0055] Figure 4 The exemplary VAD 400 relies on a reference signal 412 with a reduced user speech component. However, when the user is near an acoustically reflective surface (such as a wall or other object) or when the user's hand is near the microphone (hand behind the head, reaching for earplug 100, etc.), the user's speech may be reflected from the nearby surface and provide a second (indirect) source of the user's speech at the microphone 120. Therefore, in such cases, the second combination 408 may be less effective in rejecting user speech components. Instead, the reference signal 412 may include portions of the user's speech reflected from the nearby surface. In such cases, the VAD 400 may fail to detect speech, at least in part, because both the reference signal 412 and the main signal 410 increase as the user begins to speak, which may result in insufficient difference between the signals of the comparator 414 to determine that the user is speaking.

[0056] For example, if the user is near a wall, there may be significant reflections of the user's voice that were not rejected by the second combination 408. Furthermore, such voice energy in the reference signal 412 may also be reflected in, for example, a noise reduction system (see...). Figure 3 In the reference signal 312, this may cause the noise reduction system's adaptive processing to attempt to remove the speech.

[0057] refer to Figure 5Another exemplary VAD 500 is shown. VAD 500 is similar to VAD 400, but includes additional processing to account for the correlated energy between a first combination 506 (e.g., an MVDR beamformer) and a second combination 508 (e.g., a delay and subtraction zero-value generator) of microphone signals 504 due to a nearby reflecting surface. When a user is near an acoustically reflecting surface, indirect (reflected) speech can be substantially in phase with the user's direct speech (e.g., at low frequencies approximately 1 / 4 wavelength or less from the surface). Therefore, the second combination 508 may not reject this reflected user speech energy because it does not originate from the user's mouth direction and thus does not reach the appropriate time difference used for delay and subtraction to eliminate it. VAD 500 addresses this problem by performing addition and subtraction between a main signal 510 and a reference signal 512, and comparing the resulting sum and difference signals instead of the main and reference signals.

[0058] As described above, the first combination 506 includes the user's speech in the main signal 510. When the user is near a wall or other reflective source, lower frequency speech will be reflected into the microphone signal 504, which is not rejected (or reduced) by the second combination 508, and therefore the reference signal 512 also contains a component of the user's speech. For various frequency subbands, such as those where the reflective sources are a quarter wavelength or less away, the speech component in the reference signal 512 can be substantially in phase with the speech component in the main signal 510. Therefore, the addition of the main signal 510 and the reference signal 512 (to produce a summation signal 518) enhances the in-phase low-frequency energy, while the subtraction of one of the main signal 510 and the reference signal 512 with the other (to produce a difference signal 520) eliminates or at least significantly reduces the in-phase low-frequency energy. Thus, in the appropriate low-frequency portion of the signal spectrum, the summation signal 518 will be much larger than the difference signal 520.

[0059] In various examples, summation and difference can be complex summation and complex subtraction performed in the frequency domain (e.g., for phase and amplitude information), respectively. In other examples, summation and subtraction can be performed in the time domain.

[0060] According to various examples, summations and differences can be calculated for multiple low-frequency intervals (and various combinations thereof), and relative energy levels can be compared across one or more frequency intervals. In some examples, the VAD 500 determines the energy of each of the sum signal 518 and the difference signal 520 within the relevant frequency interval, and a low-pass filter can be applied to smooth the energy envelope. The relative level of the frequency interval is then compared to a threshold. If the threshold is exceeded, there may be a boundary interfering with the VAD beamformer. Therefore, the VAD 500 can provide an output signal 516 as logic TRUE, which can be interpreted as an indication that the user is speaking in the presence of boundary interference (near reflecting surfaces).

[0061] In various examples, several frequency ranges can be analyzed together and / or separately because the reflection path length is variable, resulting in some in-phase and out-of-phase reflections depending on the distance. For example, if a user places their hands behind their head, they are closer to the microphone array than a wall, making higher frequency ranges in-phase. A user's hand may reflect less low-frequency energy than a wall, but may reflect more high-frequency energy due to the generally closer distance. Therefore, and in some examples, for frequencies in the 200Hz to 400Hz range, a nearby wall can be detected by significant in-phase content between the main signal and the reference signal, while for frequencies in the 500Hz to 700Hz range, a user's hand can be detected by significant in-phase content between the main signal and the reference signal.

[0062] Figure 6 A method 600 for detecting user speech activity near an acoustically reflective surface is shown (such as that which can be generated by...). Figure 5 (VAD500 implementation). Method 600 receives multiple microphone signals (step 610), and combines the microphone signals according to a first combination (step 620) to provide a master signal and according to a second combination (step 630) to provide a reference signal. The first combination is configured to provide a master signal having an enhanced component representing user speech, while the second combination is configured to provide a reference signal having a reduced component representing user speech. In some examples, the first combination may be configured to provide a master signal having reduced non-speech components (such as ambient noise), while the second combination is configured to provide a reference signal having enhanced non-speech components (such as a noise reference signal (representing ambient noise)).

[0063] When the microphone signal includes reflected acoustic energy from nearby surfaces (such as walls or the user's hand, e.g., near the microphone), substantially in-phase user speech content may be present in the reference signal. This user speech content in the reference signal can cause a conventional speech activity detector to incorrectly determine that the user is not speaking, potentially leading to poor performance of other subsystems. For example, a conventional noise (or echo) reduction subsystem with adaptive filter processing (e.g., see [link to relevant documentation]). Figure 3System 300 may freeze adaptation while the user is speaking, and failure to detect user speech can cause such subsystems to begin adapting to user speech content when it shouldn't, for example, such systems typically adapt filters to noisy (or echo) content. Even when a conventional speech activity detector accurately detects speech activity, user speech content in the reference signal can cause poor performance in other subsystems if they use the reference signal as a noisy reference signal. Therefore, it is important to detect when the reference signal (erroneously) includes speech content, for example, due to nearby reflective surfaces.

[0064] As described above, for certain frequency ranges based on the distance to the reflecting surface, the speech content in the reference signal caused by the nearby reflecting surface can be in phase with the speech content in the main signal. The closer the reflecting surface, the stronger the reflection (e.g., amplitude), and the higher the frequency range where the reflection will be in phase.

[0065] Continue to refer to Figure 6 To detect in-phase user speech content in the reference signal, method 600 adds the main signal and the reference signal (step 640) to provide a summed signal and subtracts the main signal and the reference signal (calculating the difference between them) (step 650) to provide a difference signal. If significant user speech content is present in the reference signal that is in phase with the main signal, these in-phase components are added to the summed signal (enhanced) and subtracted from the difference signal (cancelled or reduced). Thus, method 600 potentially compares (step 660) the summed signal and the difference signal across various frequency ranges or frequency intervals. A sufficient difference (in energy, amplitude, etc.) between the summed signal and the difference signal at certain frequencies, ranges, or intervals means that the main signal and the reference signal contain in-phase components, which further indicate that a reflecting surface is nearby, resulting in the reference signal containing user speech components. Therefore, and as discussed above, conventional speech activity detectors may be unreliable in this case, and therefore method 600 indicates that speech activity has been detected (step 670), for example, VAD = 1.

[0066] As discussed above, other subsystems can modify their operation based on indications of speech activity, such as by freezing adaptive filters of subsystems like noise reduction, echo reduction, and / or others. In some examples, noise reduction, echo reduction, or other subsystems may cease operation when method 600 (or system 500) indicates speech activity. In various examples, when method 600 (or system 500) indicates speech activity, a main signal (such as...) can be provided... Figure 3 , Figure 4 or Figure 5The main signal (any one of 310, 410, 510) is used as the estimated speech signal so that it can be provided as the output speech signal (with or without additional processing). In other words, the lack of an indication of speech activity (or no indication of speech activity) (e.g., VAD = 0) can cause other subsystems to stop processing or stop providing the output speech signal. Therefore, in general, various examples of audio systems and methods according to those described herein may include various subsystems whose operation may depend on or not depend on binary indications of speech activity (e.g., VAD = 0 / 1), such as adjusting, changing, freezing, stopping, or starting various processes based on the output indications of the speech activity detection method 600 or system 500.

[0067] As discussed above, exemplary systems 100, 300, 400, 500 and their associated subsystems operate in the digital domain and may include analog-to-digital converters (not shown). Furthermore, the components and processes included in the exemplary systems may achieve better performance when operating on narrowband signals rather than wideband signals. Therefore, some examples may include subband filtering to allow processing of one or more subbands. For example, beamforming, nulling, adaptive filtering, signal combining (addition, subtraction), signal comparison, speech activity detection, spectrum enhancement, etc., exhibit enhanced functionality when operating on individual subbands. In some examples, subbands may be synthesized together after operation of the exemplary system to produce an output signal. In some examples, microphone signals 304, 404, 504 may be filtered to remove content outside the typical spectrum of human speech. Alternatively, exemplary subsystems may be employed to operate only on subbands within the spectrum associated with human speech and ignore subbands outside that spectrum. Additionally, while the exemplary system has been discussed with reference only to a single microphone group 120, 302, in some examples, additional microphone groups may exist, such as one group on the left and another group on the right, and other aspects and examples of the exemplary system may be applied to the additional microphone groups, and other aspects and examples of the exemplary system may be combined with the additional microphone groups.

[0068] In various examples and combinations, one or more of the above-described systems and methods can be used to capture a user's speech and isolate or enhance the user's speech relative to background noise, echoes, and other speakers. Any of the systems and methods described, and variations thereof, can be implemented at different levels of reliability based on factors such as microphone quality, microphone placement, acoustic ports, shape factor / frame design, thresholds, selection of adaptive algorithms, spectral algorithms, and other algorithms, weighting factors, window size, and other criteria that can be adapted to different applications and operating parameters.

[0069] Many (if not all) of the functions, methods, and / or components of the systems and methods disclosed herein, according to various aspects and examples, may be implemented or performed in a digital signal processor (DSP) and / or other circuitry (analog or digital) adapted to perform signal processing and other functions according to the aspects and examples disclosed herein. In addition, or as an alternative, a microprocessor, logic controller, logic circuit, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), general-purpose computing processor, microcontroller, etc., or any combination thereof, may be suitable and may include analog or digital circuitry components and / or other components relative to any particular implementation. The functions and components disclosed herein may operate in the digital domain, the analog domain, or a combination of both, and although an illustration of an analog-to-digital converter (ADC) or digital-to-analog converter (DAC) is lacking in the various figures, certain examples include ADCs and / or DACs where appropriate. Any suitable hardware and / or software (including firmware, etc.) may be configured to implement or realize the components of the aspects and examples disclosed herein, and various implementations of the aspects and examples may include components and / or functions other than those disclosed. Various implementations may include stored instructions for a digital signal processor and / or other circuitry to cause the circuitry to perform at least partially the functions described herein.

[0070] Having described several aspects of at least one example above, it should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to fall within the scope of this invention. Therefore, the foregoing description and drawings are merely exemplary, and the scope of the invention should be determined by the proper construction of the appended claims and their equivalents.

Claims

1. A method for detecting a user's voice activity, the method comprising: Receives signals from multiple microphones; The plurality of microphone signals are combined according to the first combination to generate a main signal with an enhanced response in the direction of the user's mouth; The plurality of microphone signals are combined according to the second combination to generate a reference signal with a reduced response in the direction of the user's mouth; The main signal and the reference signal are added together to generate a summation signal; The difference signal is generated by subtracting one of the main signal or the reference signal from the other. The summation signal is compared with the difference signal; as well as Based on the comparison, an output signal is provided indicating that the user is speaking.

2. The method according to claim 1, wherein the first combination is a minimum variance distortion-free response combination.

3. The method of claim 1, wherein the second combination is a combination of delay and subtraction.

4. The method of claim 1, wherein comparing the summation signal with the difference signal comprises determining at least one of the energy, amplitude, or envelope of the summation signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summation signal and the difference signal.

5. The method of claim 4, wherein comparing at least one of the energy, amplitude, or envelope of the summed signal and the difference signal comprises comparing at least one of the ratio or difference with a threshold or multiplying at least one of the energy, amplitude, or envelope by a factor and comparing the multiplied energy, amplitude, or envelope with other energies, amplitudes, or envelopes.

6. The method of claim 1, wherein comparing the summed signal with the difference signal comprises comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, the second frequency band being different from the first frequency band.

7. The method of claim 6, wherein the first frequency band includes frequencies in the range of 200Hz-400Hz, and the second frequency band includes frequencies in the range of 500Hz-700Hz.

8. The method of claim 1, further comprising processing the speech signal with an adaptive filter and modifying the adaptive filter based on the comparison.

9. An audio system, the audio system comprising: Multiple microphones; and A controller, coupled to the plurality of microphones and configured to: Receive multiple microphone signals from the multiple microphones. The plurality of microphone signals are combined according to a first combination to generate a master signal with an enhanced response in the direction of the user's mouth. The plurality of microphone signals are combined according to the second combination to generate a reference signal with a reduced response in the direction of the user's mouth. The main signal and the reference signal are added together to generate a summation signal. The difference signal is generated by subtracting one of the main signal or the reference signal from the other. The summation signal is compared with the difference signal, and Based on the comparison, an output signal is provided indicating that the user is speaking.

10. The audio system of claim 9, wherein the first combination is a minimum variance distortion-free response combination, and the second combination is a delay and subtraction combination.

11. The audio system of claim 9, wherein comparing the summed signal with the difference signal comprises determining at least one of the energy, amplitude, or envelope of the summed signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summed signal and the difference signal.

12. The audio system of claim 9, wherein comparing the summed signal with the difference signal comprises comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, the second frequency band being different from the first frequency band.

13. The audio system of claim 12, wherein the first frequency band includes frequencies in the range of 200Hz-400Hz, and the second frequency band includes frequencies in the range of 500Hz-700Hz.

14. The audio system of claim 9, wherein the controller is further configured to: process the speech signal with an adaptive filter, and modify the adaptive filter based on the comparison.

15. A non-transitory computer-readable medium having instructions encoded thereon, the instructions, when executed by a processor, causing the processor to perform a method, the method comprising: Receives signals from multiple microphones; The plurality of microphone signals are combined according to the first combination to generate a master signal with an enhanced response in the direction of the user's mouth; The plurality of microphone signals are combined according to the second combination to generate a reference signal with a reduced response in the direction of the user's mouth; The main signal and the reference signal are added together to generate a summation signal; The difference signal is generated by subtracting one of the main signal or the reference signal from the other. The summation signal is compared with the difference signal; as well as Based on the comparison, an output signal is provided indicating that the user is speaking.

16. The non-transient computer-readable medium of claim 15, wherein the first combination is a minimum variance distortion-free response combination, and the second combination is a delay and subtraction combination.

17. The non-transitory computer-readable medium of claim 15, wherein comparing the summation signal with the difference signal comprises determining at least one of the energy, amplitude, or envelope of the summation signal and the difference signal, and comparing the at least one of the energy, amplitude, or envelope of the summation signal and the difference signal.

18. The non-transitory computer-readable medium of claim 15, wherein comparing the summed signal with the difference signal comprises comparing the summed signal with the difference signal in a first frequency band and in a second frequency band, the second frequency band being different from the first frequency band.

19. The non-transitory computer-readable medium of claim 18, wherein the first frequency band includes frequencies in the range of 200Hz-400Hz, and the second frequency band includes frequencies in the range of 500Hz-700Hz.

20. The non-transitory computer-readable medium of claim 15, wherein the method further comprises: The speech signal is processed using an adaptive filter, and the adaptive filter is modified based on the comparison.

Images