Detecting accidental activation of a voice interface

Abstract

A replay detector executing in a vehicle prevents a voice interface from taking action in accordance with audio input received from a microphone when the audio input belongs to a first class of audio input, and allows the voice interface to take action in accordance with the audio input when the audio input belongs to a second class of audio input. The audio input belonging to the first class is formed by acoustic signals that have been produced by technology in an environment of the microphone. The audio input belonging to the second class is formed by acoustic signals that have been produced by at least one person in the environment.

Description

Related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 624,325, filed January 24, 2024, the contents of which are incorporated herein by reference. Background Technology

[0002] As natural language processing has matured, devices with voice interfaces have become more common. These devices typically understand and also produce speech. Therefore, it is not uncommon for a first device to receive and process speech originating from a second device rather than from a human. In such cases, the second device can be another device equipped with a voice interface. However, this second device could also be a speaker connected to a media source, such as a music source or audiovisual content.

[0003] One useful safeguard is to use a wake word. In this case, the device will ignore speech that doesn't begin with a wake word. However, the effectiveness of this safeguard depends on how rare the wake word is in the ambient speech. Because wake words can be set by the user, it's entirely possible for a user to set a wake word that frequently appears in normal speech.

[0004] For example, since users are ultimately speaking to a computer, it's not uncommon for them to simply reset the default wake word to "computer." This makes interacting with the device feel more natural. However, because the word is so common in ambient speech, there is a significant risk of accidentally triggering the computer's speech processing capabilities. Summary of the Invention

[0005] The technical problem this invention aims to solve is distinguishing between sound from a speaker and sound produced by a human. Embodiments include those in which the sound emitted by the speaker is a recording of a human voice. This typically occurs when attempting to deceive a voice interface. Embodiments also include those in which the sound emitted by the speaker is the voice of a vehicle assistant or other digital assistant. In either case, the subject matter disclosed herein reduces the likelihood that sound emitted by a speaker will trigger processing by a voice interface.

[0006] In one aspect, the invention includes an in-vehicle assistant that executes within a vehicle's infotainment system, the vehicle having a microphone providing audio input and a speaker providing audio output; a voice interface for communicating with the in-vehicle assistant; and a playback detector that prevents the voice interface from taking action based on the audio input when the audio input belongs to a first type of audio input, and allows the voice interface to take action based on the audio input when the audio input belongs to a second type of audio input. In this context, the audio input belonging to the first type is formed by acoustic signals generated by techniques already generated in the environment of the microphone, and the audio signal belonging to the second type is formed by acoustic signals generated by one or more people in the environment.

[0007] Without loss of generality, and for ease of explanation, audio inputs belonging to the first category and audio inputs belonging to the second category will be referred to herein as “abnormal” audio inputs and “non-abnormal” audio inputs, respectively. The distinction between abnormal and non-abnormal audio inputs can be understood by recognizing that the acoustic signal received by the microphone is a sound wave whose existence depends on the simultaneous presence of the human who generated the sound wave, and whose termination of presence causes the acoustic signal at the microphone to terminate after a time interval, which depends on the speed of sound and the distance between the microphone and the location where the human was previously located.

[0008] In contrast, non-abnormal audio input is formed by acoustic signals whose presence is independent of the presence of humans. Such acoustic signals can exist even without the presence of any humans. Examples of such acoustic signals include human recordings, the voice output of digital assistants, or the sound of broadcast programs.

[0009] The distinction between anomalous and non-nominal audio input can also be understood by recognizing that acoustic signals at a microphone can be generated in two ways: one as a result of the conversion of electrical energy into mechanical energy, and the other as a result of the conversion of chemical energy into mechanical energy. Examples of the former include playback from a loudspeaker. Examples of the latter include human voices emanating from humans using chemical energy derived from human metabolic processes and converted into mechanical energy, which performs mechanical work on the diaphragm to provide a fluid velocity sufficient to initiate vibration of the vocal cords, and using muscles to cause time-varying tension in the vocal cords, which similarly convert chemical energy into mechanical energy to perform mechanical work on the vocal cords. The former is considered a "technology-generated acoustic signal," while the latter is not.

[0010] An example of a technology-generated acoustic signal is an acoustic signal formed by reproducing a recorded human voice. In such a case, the acoustic signal incident on the microphone is the result of the following conversions: chemical energy is converted into mechanical energy by humans; this mechanical energy is converted into electrical energy for storage on a recording medium; the electrical energy is converted back into mechanical energy during playback via a loudspeaker; and the mechanical energy is converted back into electrical energy at the microphone.

[0011] Another example of acoustic signals generated by technology is the acoustic signal formed by the spoken words of an in-vehicle assistant or digital assistant. In this case, the electrical energy associated with the electrical signal representing speech is converted into mechanical energy by the speaker, and this mechanical energy is then converted back into electrical energy by the microphone.

[0012] An example of non-technically generated acoustic signals (which in turn would form non-abnormal audio signals) is acoustic signals generated by humans in the microphone's environment. In such cases, chemical energy from human metabolism is converted into mechanical energy, which is then incident on the microphone to be converted into electrical energy.

[0013] This invention relies on the discovery that there are telltale features in acoustic signals, which can ultimately be used by machine learning systems to examine audio signals to distinguish between the two types of acoustic signals.

[0014] As used in this article, "acoustic signal" refers to sound waves, while "audio signal" refers to signals that occupy the frequency bands that humans can hear.

[0015] The embodiments include those in which the playback detector has been trained using machine learning techniques. These embodiments include: those in which the playback detector has been trained to distinguish between anomalous and non-nominal audio inputs; those in which the playback detector has been trained to distinguish between spectrograms representing anomalous and non-nominal audio inputs; those in which the playback detector has been trained to distinguish between the output of a general signal processing method representing anomalous audio input and the output of a general signal processing method representing non-nominal audio input; those in which the playback detector has been trained based on training audio signals obtained using acoustic transducers in a vehicle (including speakers and microphones); and those in which the playback detector has been trained using the impulse response of the vehicle.

[0016] Other embodiments include a signal conditioner that modulates the audio input before it reaches the voice interface from the microphone. These embodiments include: those in which the playback detector receives the audio input after it has been modulated; those in which the playback detector receives the audio input before it has been modulated; and those in which the playback detector receives the audio input directly from the microphone.

[0017] Other embodiments also include a switch that toggles between providing conditioned audio input to the playback detector and providing unconditioned audio input to the playback detector.

[0018] Other embodiments include a switch having a state that prevents the voice interface from receiving audio input from a microphone, wherein a playback detector causes the switch to be in this state when the microphone is receiving abnormal speech, and causes the switch to be out of this state when the microphone is receiving natural speech. These embodiments include: embodiments in which the switch is configured to allow the voice interface to receive audio input until the playback detector has determined that the audio input is abnormal (at which point it disconnects the switch); and embodiments in which the switch is configured to block the voice interface from receiving audio input until the playback detector has determined that the audio input is not abnormal.

[0019] In one aspect, the present invention includes a digital assistant, a voice interface, and a playback detector that prevents the voice interface from taking action based on the audio input when the acoustic input is abnormal speech, and allows the voice interface to take action based on the audio input when the audio input is non-abnormal audio input.

[0020] In another aspect, the invention is characterized by a method comprising: detecting that a first audio input obtained from a microphone is an anomalous audio input; preventing a voice interface from taking action based on the first acoustic input; detecting that a second audio input obtained from the microphone is a non-abnormal audio input; and allowing the voice interface to take action based on the second acoustic input.

[0021] The aforementioned methods include those practices where the microphone is located in a vehicle.

[0022] These and other features of the invention will become apparent from the following detailed description and accompanying drawings, in which: Attached Figure Description

[0023] Figure 1 An exemplary architecture for a system used to reduce unintended triggering of a voice interface is shown. Detailed Implementation

[0024] Figure 1A vehicle 10 is shown with an infotainment system 12 that receives audio input 14 from a microphone 16 and provides audio output 18 via a speaker 20.

[0025] The infotainment system 12 is equipped with an in-vehicle assistant 22. A voice interface 24 between the microphone 16 and the in-vehicle assistant 22 establishes two-way voice communication with the occupants of the vehicle 10.

[0026] The first switch 26 interrupts the signal path between the microphone 16 and the voice interface 24. Whether switch 24 is open or closed determines whether the audio input 14 received by the voice interface 24 has ever reached the voice interface 24.

[0027] The playback detector 28 controls the state of the first switch 26. The playback detector 28 controls this state by classifying the audio input 14 as abnormal or non-abnormal.

[0028] In some embodiments, the first switch 26 remains off by default, such as Figure 1 As shown. Upon receiving an audio signal, the playback detector 28 classifies the audio signal as either abnormal or normal. If the audio signal has been classified as abnormal, the playback detector 28 takes no action to change the state of the first switch 26. Conversely, if the audio signal has been classified as normal, the playback detector 28 sends a first control signal 30 to close the first switch 26. The first switch 26 remains closed until the voice interface 24 detects the end of the voice in the audio signal, at which point the first switch 26 is opened again.

[0029] One drawback of the aforementioned embodiments is that some delay is unavoidable in the classification process. In an alternative embodiment, the first switch 26 is changed to remain closed by default and is only opened after the playback detector 28 has classified the audio signal as an abnormal audio signal.

[0030] In either case, the voice interface 24 then provides the voice to the vehicle assistant 22 in a form suitable for processing by the vehicle assistant 22. If necessary, the vehicle assistant 22 provides data to the voice interface 24, which then uses the data to construct a suitable voice for transmission via the speaker 20.

[0031] In some embodiments, a signal conditioner 32 configured to receive audio input 14 from microphone 16 conditions the audio input 14 before sending it to voice interface 24. Examples of signal conditioners 32 include those that remove background noise (e.g., road noise) from the audio input 14, and those that enhance various psychoacoustic metrics in the audio input 14. For ease of explanation, the signal leaving the signal conditioner 32 should be referred to as “audio input 14” even if some characteristics of the signal have changed slightly. Where necessary, references will be made to “upstream” and “downstream” signals from the signal conditioner 32.

[0032] Receiving the audio input 14 is useful for the playback detector 28 to perform the function of classifying the audio input 14. In some cases, receiving the raw audio input 14 before it is processed by the signal conditioner 32 results in a lower error rate. In other cases, the reverse is true. For example, if the signal conditioner 32 happens to suppress the artifacts on which the playback detector 28 relies for classification during the performance of its function, then receiving the audio input 14 upstream of the signal conditioner 34 results in a lower error rate. On the other hand, if the signal conditioner 32 happens to make the artifacts on which the playback detector 28 relies for classification more prominent during the performance of its function, then receiving the audio input 14 downstream of the signal conditioner 32 results in a lower error rate. To provide additional flexibility to the playback detector 28, the second switch 34, in response to the second control signal 36, allows the playback detector 28 to be connected upstream or downstream of the signal conditioner 32.

[0033] The replay detector 28 implements a machine learning system 38, which has been trained using training signals including anomalous speech from various combinations of speakers and microphones, and natural speech from various combinations of humans and microphones. Based on this training data, the machine learning system 38 becomes increasingly adept at identifying subtle artifacts present in anomalous speech but not in natural speech. This results in a machine learning system 38 that classifies between natural and anomalous speech using a low error rate.

[0034] In a preferred embodiment, the training signal is converted into a spectrogram. This is particularly useful because it allows for the full utilization of conventional image classification algorithms for classifying speech signals rather than conventional images. The spectrograms obtained from the training data are either non-abnormal spectrograms representing non-abnormal audio inputs, or anomalous spectrograms representing anomalous audio inputs. In this case, signal processing methods can be developed to enhance the features that distinguish between these two types of spectrograms, and then such signal processing techniques can be applied to the incoming audio input.

[0035] An example of such a signal processing technique is "mean spectral subtraction," where the mean is determined solely based on the training data from the first class. The resulting mean spectrogram is then subtracted from all other spectrograms (including both the training and test spectrograms used for both anomalous and non-anomalous spectrograms). The resulting classifier is thus tuned to detect anomalous audio input and label anything that is not non-anomalous as "abnormal." Therefore, this type of classifier is robust to unforeseen spoofing attacks. After all, the classifier does not rely on prior knowledge of the characteristics of such spoofing attacks.

[0036] In other embodiments, the replay detector 28 implements different methods for rejecting abnormal speech. These methods include those that do not require machine learning, such as extracting computed features from the audio input.

[0037] Implementations of the replay detector 28 that rely on machine learning include those that rely on deep learning (such as deep learning performed by deep neural networks) and those that rely on classical machine learning methods (such as Markov models, support vector machines, and Gaussian mixture models).

[0038] In the context of vehicle 10, the positions and corresponding properties of speaker 20 and microphone 16 are known in advance. Additionally, the impulse response of the vehicle is also known in advance. In such cases, it is particularly useful for the training signal to include a signal obtained from speech within vehicle 10. In those cases that include signal conditioner 32, it is useful to provide a training signal that incorporates the effects of signal conditioner 32.

[0039] Having described the invention and its preferred embodiments, the following is claimed as novel content and protected by an authorized patent.

Claims

1. An apparatus, the apparatus comprising: A vehicle assistant (22) is executed in an infotainment system (12) of a vehicle (10) having a microphone (16) providing audio input (14) and a speaker (20) providing audio output (18); a voice interface (24); and a playback detector (28) that prevents the voice interface from taking action based on the audio input when the audio input belongs to a first type of audio input, and allows the voice interface to take action based on the audio input when the audio input belongs to a second type of audio input, wherein the first type of audio input includes audio input formed by acoustic signals generated by a technique already generated in the environment of the microphone, and wherein the second type of audio input includes audio input formed by acoustic speech signals generated by at least one person in the environment, wherein the at least one person includes vocal organs, and wherein the vocal organs generate the acoustic speech signals when the at least one person is in the environment.

2. The apparatus according to claim 1, wherein, The playback detector has been trained by machine learning to distinguish between audio inputs belonging to the first type of audio input and audio inputs belonging to the second type of audio input.

3. The apparatus according to claim 1, wherein, The playback detector includes a deep neural network that has been trained to distinguish between audio inputs belonging to the first type of audio input and audio inputs belonging to the second type of audio input.

4. The apparatus according to claim 1, wherein, The playback detector includes a classifier that has been trained to distinguish between a first spectrogram and a second spectrogram, wherein the first spectrogram is a spectrogram belonging to a first class of spectrograms, and the second spectrogram is a spectrogram belonging to a second class of spectrograms, wherein the first class of spectrograms includes spectrograms from audio inputs belonging to the first class of audio inputs, and wherein the second class of spectrograms includes spectrograms from audio inputs belonging to the second class of audio inputs.

5. The apparatus according to claim 1, wherein, The vehicle includes an acoustic transducer, the speaker and the microphone are in the acoustic transducer, and the playback detector includes a classifier that has been trained based on training audio signals obtained using the acoustic transducer in the vehicle.

6. The apparatus according to claim 1, wherein, The playback detector includes a classifier that has been trained using impulse responses from the cabin in the vehicle to distinguish audio inputs belonging to the first class from audio inputs belonging to the second class.

7. The apparatus according to claim 1, wherein, The playback detector compares the acoustic features obtained from the audio input with the features calculated from the training set.

8. The apparatus according to claim 1, wherein, The playback detector includes a classifier that has been trained based on the acoustic characteristics of the cabin in the vehicle.

9. The apparatus of claim 1 further includes a signal conditioner (32) that adjusts the audio input before it reaches the voice interface.

10. The apparatus according to claim 1, wherein, The playback detector receives the audio input after the audio input has been adjusted.

11. The apparatus according to claim 1, wherein, The playback detector receives the audio input directly from the microphone.

12. The apparatus of claim 1 further includes a switch (34) that switches between providing conditioned audio input to the playback detector and providing unconditioned audio input to the playback detector.

13. The apparatus of claim 1, further comprising a switch (26) having a state that prevents the voice interface from receiving the audio input from the microphone, wherein, The playback detector causes the switch to be in the state in response to receiving an audio input of the first type, and causes the switch to be out of the state in response to receiving an audio input of the second type.

14. A method, the method comprising: Receive the first audio input from the microphone; It is determined that the first audio input belongs to the first type of audio input; Prevent the voice interface from taking action based on the first audio input; Receive a second audio input from the microphone; determine that the second audio input belongs to a second type of audio input; and allow the voice interface to take action based on the second acoustic input, wherein the first type includes audio input formed by acoustic signals generated by a technique already generated in the environment of the microphone, and wherein the second type includes audio input formed by acoustic speech signals generated by at least one person in the environment.

Images