Environmental aware voice assist devices and related systems and methods

Abstract

The invention relates to environment-aware voice assist devices and related systems and methods. An appliance may include a microphone transducer, a processor, and a memory storing instructions. The appliance is configured to receive an audio signal at the microphone transducer and detect an utterance in the audio signal. The appliance is further configured to classify a speech pattern based on the utterance. The appliance is further configured to determine a condition of an environment of the appliance. The appliance is further configured to select at least one of a playback volume or a voice output mode of a plurality of voice output modes based on the classification and the condition of the environment of the appliance. The appliance is further configured to adjust a playback volume and / or mode of playback voice according to the voice output mode. The appliance may be configured to synthesize speech according to the speech output pattern, or to modify synthesized speech according to the speech output pattern.

Description

[0001] Related Applications Citations

[0002] This application is a divisional application of the invention patent application with Chinese national application number 202010971798.X, application date September 16, 2020, and invention name “Environmentally-aware voice-assisted equipment and related systems and methods”. Technical Field

[0003] This patent application and the subject matter disclosed herein (collectively, the present "disclosure") generally relate to voice assistant devices, including voice assistant devices capable of synthesized speech playback, and related systems and methods. More specifically, but not exclusively, the present disclosure relates to systems, methods, and components for adapting synthesized speech or playback of synthesized speech to cues observed in a listening environment in order to improve the intelligibility of the synthesized speech and the user experience. Background Art

[0004] Audio appliances are increasingly able to perform multiple tasks in response to commands spoken by the user, and often already interact with machines such as, for example, smart speakers, computers, mobile devices, navigation systems, cars, and other computing environments more naturally than, for example, using tactile or key inputs. In principle, such appliances collect sound from an acoustic scene, extract information (e.g., commands) from the collected sound, and respond to the extracted information.

[0005] When such an audio appliance performs a task, the result of the task may include text, which may be synthesized into speech by the audio appliance. The synthesized speech may be output to the user, for example, through a speaker, thereby providing an output that is perceptible to the user. Various conditions associated with the user's listening environment and various aspects of the synthesized speech may adversely affect the quality or intelligibility of the synthesized speech as perceived by the user.

[0006] As used herein, the term "intelligibility" refers to a measure of the perceived quality of sound containing speech, such as whether speech can be heard by a listener or to what extent speech can be understood by a listener. The intelligibility of a given sound output may vary across different listening environments. Summary of the invention

[0007] The concepts, systems, methods, and devices disclosed herein relate to synthesizing speech or modifying already synthesized speech based on acoustic and non-acoustic cues that reflect characteristics of the user's listening environment. Such cues may include, among other things, characteristics of the user's voice used to invoke a voice assistant. For example, when an audio device is in a noisy environment, the user's speech used to invoke a voice assistant may exhibit a so-called "Lombard effect." In response, the audio device may synthesize speech (or modify the synthesized speech) to exhibit characteristics of the Lombard effect, thereby improving the intelligibility of the synthesized speech in that environment.

[0008] In some aspects, the disclosed concepts relate to improving the intelligibility of previously synthesized speech based on observed characteristics of the listening environment. Such characteristics may include, for example, the user's intonation and speech rhythm, as well as other observable characteristics of the user's speech pattern. These and other characteristics of the listening environment may include direct or indirect observation of reverberation, background noise, voice quality, distance from the speaker, etc. Other characteristics (including non-acoustic cues) that suggest the user's listening environment may include, for example, the time of day, the volume setting on the audio instrument, and the location of the audio instrument.

[0009] Some disclosed concepts relate to systems, methods, and components for synthesizing speech that can be understood in the listening environment in which the speech is played. For example, some disclosed audio instruments can use selected clues that suggest the environment to select a model or parameters for synthesizing speech customized for a given listening environment.

[0010] According to one aspect, an audio device has an audio acquisition module having a microphone transducer and configured to receive an audio signal. The device also includes one or more other functional components. For example, the device may include a speech classifier configured to classify the speech pattern of the received speech. Similarly, the device has a decision-making component configured to identify conditions associated with a listening environment and select a speech output mode from a plurality of speech output modes. The audio device also has an output component configured to output a synthesized speech according to the speech output mode.

[0011] The decision component may select one or more speech synthesis parameters corresponding to the speech output mode. The decision component may also or alternatively select the playback volume. When combined in the speech synthesis model, the one or more speech synthesis parameters may make the speech pattern of the synthesized speech match the speech pattern of the utterance. In other cases, when combined in the speech synthesis model, the one or more speech synthesis parameters may make the speech pattern of the synthesized speech different from the speech pattern of the utterance. In some cases, the decision component may select the speech synthesis model from a plurality of speech synthesis models corresponding to the speech output mode.

[0012] The audio device also has a speech synthesizer configured to synthesize speech. Such synthesis may include selecting a speech synthesis model from a plurality of speech synthesis models, selecting one or more speech synthesis parameters applicable to a given speech synthesis model based on a selected speech output mode, a selected playback volume, or both, or both.

[0013] The speech synthesizer may be configured to receive one or more speech synthesis parameters and text to be synthesized into speech. The synthesizer may also be configured to provide one or more speech synthesizer parameters and text as input to a speech synthesis model. The synthesizer may be configured to generate synthesized speech from text using a speech synthesis model, for example, according to one or more speech synthesis parameters. The appliance may have an output component configured to output synthesized speech.

[0014] In some cases, the decision component may select one or more speech modification parameters corresponding to the speech output mode. The audio appliance may also have a speech modification component configured to modify the synthesized speech according to the selected one or more speech modification parameters, according to the selected playback volume, or both, before outputting the modified synthesized speech.

[0015] The speech classifier may be configured to classify the speech mode of the utterance as a whispered mode, a normal mode, or a Lombard effect mode. For example, the speech classifier may be configured to classify the speech mode of the utterance as a whispered mode, a normal mode, or a Lombard effect mode according to at least one of the following: pitch, energy content, number of formants, spectral tilt, speech rate, or a combination thereof.

[0016] The conditions associated with the listening environment that the decision component is configured to identify may include at least one of the following: acoustic cues inferred from the audio signal, non-acoustic cues, or a combination thereof. The acoustic cues may include at least one of the following: background noise received by a microphone transducer, direct-to-reverberation sound energy ratio, signal-to-noise ratio, echo coupling residual, distance to a speaker, voice quality metrics, room characteristics, or a combination thereof. The non-acoustic cues may include at least one of the following: time of day, location type, appliance mode, user profile, location layout, and acoustic profile of the location.

[0017] The output component may include a speaker and a digital signal processing component configured to process the synthesized speech for output by the speaker.

[0018] The appliance may include an input processor configured to receive an audio signal from a microphone transducer, process the audio signal, and invoke a speech recognition task on the processed audio signal. The received text may correspond to a response generated according to the speech recognition task.

[0019] The instrument may include one or more additional microphone transducers positioned at one or more locations on the audio instrument. The one or more additional microphone transducers may be configured to receive sound and provide corresponding audio signals to the input processor.

[0020] In other aspects, the concepts disclosed herein may relate to an apparatus having a microphone transducer, a processor, and a memory storing instructions. When executed by a processor, the instruction may cause the apparatus to receive an audio signal at the microphone transducer. The instruction may also cause the apparatus to detect speech in the audio signal, and analyze the speech to classify speech patterns based on the speech. The instruction may also cause the apparatus to determine conditions associated with a listening environment. For example, the condition may include one or more acoustic cues and one or more non-acoustic cues from an audio signal, wherein each non-acoustic cue corresponds to a condition of the environment of the apparatus. The instruction may also cause the apparatus to select a speech output mode from a plurality of speech output modes based on classification, one or more acoustic cues, and one or more non-acoustic cues. In addition or alternatively, the instruction may cause the apparatus to select, for example, a playback volume of a synthesized speech. The instruction may also cause the apparatus to output a synthesized speech according to a speech output mode, a selected playback volume, or both.

[0021] Some instructions may cause the appliance to classify the utterance according to one or more features of the utterance to classify the speech mode. The features may include, for example, but not limited to, pitch, number of formants, spectral tilt, speech rate, energy content, or a combination thereof. The instructions may cause the appliance to classify the speech mode of the utterance as a whisper mode, a normal mode, or a Lombard effect mode according to the one or more features.

[0022] Some instructions may cause the appliance to receive an audio signal from a microphone transducer and process the audio signal. The instructions may also cause the appliance to request speech recognition of the processed audio signal. In response to requesting speech recognition, the appliance may receive text that may correspond to a response generated based on the recognized speech.

[0023] The instruction may also cause the device to synthesize text into speech. In some cases, the instruction may cause the device to synthesize text into speech according to the speech output mode. For example, the instruction may cause the device to select one or more speech synthesis parameters, and generate synthesized speech according to the speech synthesis model and the selected one or more speech synthesis parameters. For another example, the instruction may cause the device to select a speech synthesis model from a plurality of speech synthesis models, and generate synthesized speech according to the selected speech synthesis model.

[0024] In other cases, the instructions may cause the appliance to modify the synthesized speech according to the speech output mode. For example, the instructions may cause the appliance to select one or more speech modification parameters based on the speech output mode, and modify the synthesized speech according to the one or more speech modification parameters and output the modified synthesized speech. According to another aspect, the instructions may cause the appliance to modify the volume of speech playback according to acoustic cues of the environment (e.g., independent of the speech synthesizer).

[0025] Also disclosed are associated methods, and tangible, non-transitory computer-readable media including computer-executable instructions that, when executed, cause a computing environment to implement one or more methods disclosed herein. Also disclosed are digital signal processors embodied in software, firmware, or hardware and suitable for implementing such instructions.

[0026] The foregoing and other features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Referring to the drawings, wherein like numerals refer to like parts throughout the several views and the specification, various aspects of the disclosed principles are illustrated by way of example and not by way of limitation.

[0028] Figure 1 A schematic block diagram of a first system for adapting playback of a voice assistant device is shown.

[0029] Figure 2 A schematic block diagram of a second system for adapting playback of a voice assistant device is shown.

[0030] Figure 3 A schematic block diagram of a third system for adapting playback of a voice assistant device is shown.

[0031] Figure 4 An example of the relationship between environmental parameters and speech classification is shown.

[0032] Figure 5 An example of control logic that may be used by a system for adapting playback of a voice assistant device to an environment is shown.

[0033] Figure 6 An example of control logic that may be used by a system for adapting playback of a voice assistant device to an environment is shown.

[0034] Figure 7 A voice assistant device with a circular array of microphones exposed to sound from two directions is shown, and a graph of the microphones' response to sound from each direction is also schematically shown.

[0035] Figure 8 A block diagram is shown, illustrating aspects of an audio instrument.

[0036] Fig. 9 A block diagram of a computing environment suitable for implementing the disclosed methods or control logic is shown. DETAILED DESCRIPTION

[0037] Various principles related to adapting synthesized speech or playback of synthesized speech to cues observed in a user's listening environment are described below. In one aspect, the disclosed principles select synthesized speech modifications to improve intelligibility in a listening environment. As an illustrative example only, an audio device can detect speech, classify speech patterns based on features of the speech, and identify conditions present in and associated with the listening environment. Based on the observed features, the audio device can adjust the synthesis of speech, modify the synthesized speech, and / or change the playback volume. Such modifications can make the output of the synthesized speech understandable in challenging environments, comfortable to listen to in other environments, or generate whispered speech in quiet and specific environments, each of which can improve the user's experience in different listening environments.

[0038] In one aspect, certain aspects of the disclosed principles relate to classifying the user's speech. For example, the classification can be performed according to a plurality of speech patterns and conditions detected or identified associated with the user's listening environment. Synthesized speech can be generated according to speech patterns and environmental conditions, or previously generated synthesized speech can be modified to achieve a desired level of intelligibility. For example, the synthesized speech can be modified linearly, such as by increasing or decreasing its sound level / volume, or the synthesized speech can be modified with advanced signal processing techniques (e.g., in the time domain and the spectral domain) to simulate a low voice or Lombard effect speech, as described below. That is, the description of the combination of specific device configurations and method actions herein is only a specific example of an intended system selected as a convenient illustrative example of the disclosed principles. One or more disclosed principles can be combined in various other systems to implement any of the various corresponding system features.

[0039] Thus, systems having properties other than those of the specific examples described herein may embody one or more principles of the present disclosure and may be used in applications not described in detail herein. Such alternative aspects are therefore also within the scope of the present disclosure.

[0040] I. Overview

[0041] Methods for improving the perceived intelligibility and comfort of synthesized speech may take into account characteristics of the user's speech, as well as acoustic and non-acoustic cues that indicate the user's listening environment. For example, those characteristics and cues may indicate whether a given listening environment is noisy or quiet. Such indications may be used to infer conditions surrounding the audio instrument, such as a particular location or room layout, room class, or time of day.

[0042] The disclosed principles may be implemented as, for example, properties that affect synthesized speech during or after synthesis, so that the output of synthesized speech is understandable to a user in a given observed listening environment. Similarly, the disclosed principles may be implemented as properties that affect synthesized speech, so that the output is suitable for the user's listening environment, while making the output understandable. For example, in a quiet environment, the output synthesized speech may be whispered by an audio device. In a noisy environment, the output synthesized speech may be loud enough to cover background noise and be heard, while being different from normal speech in a way that makes the speech more understandable, as described below.

[0043] In some cases, adjusting the volume or level of the output speech may be sufficient to maintain the comfort and intelligibility of the speech. That is, additional or alternative methods for adapting speech to the observed listening environment are also described to improve the user experience, such as by improving speech intelligibility.

[0044] Methods for adjusting synthesized speech and / or speech playback may include detecting conditions (or features) associated with a given listening environment, such as, but not limited to, background noise levels, acoustic maps of rooms, and / or time of day. These methods may also or alternatively detect features of a user's voice pattern, such as a low voice, a whispered narration, a "normal" voice, a loud, prominent voice, or shouting. The detected conditions and the user's voice pattern may be input to a decision-making component, which makes a decision on playback volume, a voice output mode for outputting synthesized speech, or both. Before outputting to the user, the selected voice output mode may determine how to synthesize speech or how to modify the synthesized speech, so that the output synthesized speech is more easily understood, comfortable, and sufficient for the user's listening environment in terms of perception. Therefore, the intelligibility of the output synthesized speech is retained or improved in a manner that does not merely change the volume of the speech or as a supplement to changing the volume of the speech. In some cases, the user's voice pattern may match the synthesized speech, while in other cases, the voice pattern of the output synthesized speech may be different from the user's voice pattern.

[0045] Further details of the disclosed principles are as follows. Part II discusses various aspects and modes of human speech. Part III describes principles related to a method in which a decision component may select one or more speech synthesis parameters that, when applied by a speech synthesizer, may change the way a speech synthesis model generates synthesized speech (when compared to a "normal" output). Part IV describes principles related to a different method in which a decision component may select a specific speech synthesis model from a plurality of speech synthesis models for use by a speech synthesizer. Part V describes principles related to yet another method in which a decision component may select one or more speech modification parameters, volume mappings, or speech modification parameters and volume mappings that a speech modification component may use to modify already synthesized speech to improve the intelligibility of synthesized speech in a listening environment and / or to adapt playback to an environment by increasing or decreasing the volume. Part VI discloses various examples of control logic that may be used in the disclosed method. And, Part VII discloses principles related to an audio device and computing environment that are applicable to implementing the disclosed improved technology for synthesized speech intelligibility.

[0046] Other related principles are also disclosed. For example, the following describes a machine-readable medium containing instructions that, when executed, cause a processor, such as a computing environment, to perform one or more disclosed methods. These instructions may be embedded in software, firmware, or hardware. In addition, the disclosed methods and techniques may be executed by various forms of processors or controllers (such as in software, firmware, or hardware).

[0047] II. Voice Mode

[0048] Human speech can be characterized by one or more metrics. Examples of these metrics include, but are not limited to, pitch, sound level or energy content, spectral tilt, speech rate, and formants. Pitch refers to the relative high or low pitch perceived by the ear. For human speakers, pitch depends on the number of vibrations per second produced by the speaker's vocal cords. Pitch can be measured (or quantified) by frequency. A given speaker can produce a range of pitches.

[0049] The energy content in a sound field (e.g., speech) is related to the sound pressure, particle velocity, and density of the medium through which the sound propagates. The energy content can be quantified using energy units such as joules. Sound level or "sound level" is the logarithmic ratio of the energy content to a reference energy content and is usually expressed in dimensionless units such as decibels (dB).

[0050] Spectral tilt refers to the relationship between the frequencies of speech and the sound level or energy content of each frequency. When plotted, for example, in decibels per hertz (dB / Hz), the spectral tilt is the slope of the curve. In normal human speech, the higher frequencies in speech typically have lower energy content than the lower frequencies, so spectral tilt is typically negative over the human vocalization band.

[0051] Speech rate may refer to whether a person speaks fast or slow, for example, as measured by words per minute (wpm). Normal speech rates may range from about 120 wpm to about 200 wpm, such as 130 wpm, 140 wpm, 150 wpm, and 160 wpm. Speech rates may vary, for example, based on the speaker's culture, gender, and / or geographic location. Likewise, the speech rate of a given individual may also vary, for example, based on the subject, emotional state, fluency, expertise, or audience.

[0052] A formant refers to the concentration of acoustic energy around a specific frequency in the sound that carries the speech, and corresponds to a resonance in the vocal tract. Human speech includes several formants. Each formant occurs at a different frequency, about one per 1,000 Hz band. That is, in human speech, formants occur at intervals of about 1,000 Hz.

[0053] Compared to normal or neutral speech, whispered speech lacks pitch and has no periodic excitation of the harmonic structure. Whispered speech also has higher formant frequencies than normal speech, but lacks layer formants. Whispered speech also has a flatter spectral slope than normal speech and has lower energy content.

[0054] On the other hand, Lombard effect speech refers to a generally unintentional human response to speaking in a noisy environment, wherein the speech is modified to improve intelligibility. Compared with normal speech, Lombard effect speech is characterized by increased pitch, slower speech rate, and increased sound level or energy content. Lombard effect speech also exhibits a reduced spectral tilt caused by the energy content shifting toward high frequencies. The energy content can also be more concentrated in terms of formant frequencies relative to normal speech.

[0055] As used herein, a "listening environment" or "environment" refers to the location of the audio device to which the speakers on the audio device will be directed to play the synthesized speech to the user. Generally speaking, the listening environment will be the same as the recorded environment in which the user speaks to interact with the voice assistant application on the audio device.

[0056] III. Example 1

[0057] Figure 1An example of a first system 10 for adapting the playback of a voice assistant device to an environment to retain or improve the intelligibility of synthesized speech or other played-back speech is shown. The system 10 may include an audio device 100. The audio device 100 may be embodied as, for example, a computing device. In some aspects, the audio device is specifically a mobile communication device, such as a smart phone or tablet, or a personal or home assistant device, such as a smart speaker. The audio device 100 may include a voice assistant application (not shown) that is configured to listen to, understand and respond to a user's spoken requests or commands. Alternatively, the voice assistant application may reside on a different device, such as a network-connected device, or the voice assistant application may be distributed between multiple network-connected devices.

[0058] The appliance 100 may be configured to listen for and respond to speech activation commands, for example, using an activation component (not shown). The activation component (which may be a component in the input unit 120) may listen for activation commands in an incoming audio signal. For example, the activation component may recognize speech in the audio signal and may analyze the speech to detect one or more specific keywords or key phrases, such as those that may be used to trigger a voice assistant application.

[0059] In response to detecting the activation command, the voice assistant application may cause the device to listen for user speech. Upon receiving the user's speech, the voice assistant application may send an audio signal containing the speech to the speech recognition component. The voice assistant application may then perform a task or respond to a request in response to the command or request detected in the recognized speech.

[0060] The input unit 120 may include an input digital signal processor (DSP) 132. The input unit 120 may include an audio acquisition module (AAM) (not shown) having a microphone transducer (e.g., microphone 104-1) configured to capture ambient sounds. The ambient sounds may include the user's speech 102. The input DSP 132 may process, filter, or otherwise condition or prepare the captured representation of the ambient sounds for speech recognition tasks, and may also provide the resulting audio signal to the speech classifier 126. For example, the input DSP 132 may remove noise, perform echo cancellation, and remove reverberation from the audio signal.

[0061] The speech classifier 126 may analyze, extract, measure or otherwise determine one or more features of the speech in order to classify the speech pattern of the speech. For example, the speech classifier may classify the speech pattern into a whispered pattern, a normal pattern or a Lombard effect pattern. Other patterns may also be identified, such as a soft pattern between a whispered speech and a normal speech, and / or a loud pattern between a normal speech and a Lombard effect speech. In one example, the speech classifier 126 may compare, for example, the pitch of the speech with a pitch threshold or range, compare the spectral tilt of the speech with a spectral tilt threshold or range, and / or compare the energy content of the speech with an energy content threshold or range. For example, if the speech lacks pitch and / or has less energy than a threshold, the speech classifier 126 may classify the speech as a whisper. If the spectral tilt of the speech is flatter relative to the threshold spectral tilt, and the pitch of the speech is above the pitch threshold, the speech classifier 126 may classify the speech as a Lombard effect pattern. Those skilled in the art will formulate that a "speech" may include one or more words, phrases or sentences, and may include keywords that can trigger the voice assistant of the system 10. Other classification methods may be used. For example, the speech classifier 126 may include a neural network component that may receive information about one or more aspects of the utterance, conditions in the environment, or a combination thereof, and then output a speech pattern classification based on the received information.

[0062] Although in Figure 1 10, but in some aspects, the speech classifier 126 may be remote from the appliance 100. For example, the speech classifier 126 may be a component of a cloud-based speech classification system or a component of a remote speech recognition system (e.g., a remote automatic speech recognition (ASR) system 110). Although the ASR system 110 is depicted as being remote from the device 10, some or all automatic speech recognition operations may be performed on the device 10. Therefore, the ASR system 110 may be incorporated on the device 10.

[0063] The processing performed by the input DSP 132 may also include extracting acoustic cues 136 from the current listening environment. For example, the acoustic cues 136 may be extracted from ambient sounds present in the audio signal from one or more microphones. Examples of acoustic cues may include background noise level, direct-to-reverberation sound energy ratio, signal-to-noise ratio, echo coupling residual, quality measures of speech, output sound level of audio (e.g., music) that has been played back by an instrument, and distance from a human speaker.

[0064] The input unit 120 may also include a decision component 134. The decision component 134 may be configured to receive the recognized speech pattern from the speech classifier 126, and may receive or retrieve acoustic cues 136. Likewise, the decision component may receive or retrieve other cues indicative of or associated with the listening environment or user preferences, such as non-acoustic cues 138. The decision component 134 may identify conditions associated with the listening environment of the audio instrument based on the received and / or retrieved information.

[0065] Non-acoustic cues 138 may include information about the environment and user preferences that are unrelated to the sounds in the listening environment. Examples of non-acoustic cues may include the time of day, the location of the audio appliance (e.g., in a room or vehicle), the type of room the audio appliance is in (e.g., a kitchen, bedroom, or living room), a known layout of the room, or metadata related to the environment. Some audio appliances (e.g., home assistant devices) may have an acoustic profile of the room in which the audio appliance is operating, and may have information about the distances between various microphones and human speakers that the appliance has access to.

[0066] The decision component 134 may determine, for example, that the listening environment is a very quiet environment where whispered speech is expected. This may be determined based on, for example, receiving an indication of whispered speech patterns from a speech classifier, acoustic cues of low-level background noise; non-acoustic cues, including the time of day, the location of the bedroom, the relatively small distance between the audio instrument and the user (e.g., less than an arm's length); or any combination of these.

[0067] As another example, the decision component 134 may determine that the listening environment is a very noisy environment, in which Lombard effect speech and increased volume will improve the intelligibility of the synthesized speech. This may be based on, for example, receiving an indication of a Lombard effect speech pattern from a speech classifier, an acoustic clue indicating a high level of background noise; non-acoustic clues, including the time of day associated with high level background noise (e.g., 7 p.m.), a relatively large distance between the audio instrument and the user (e.g., farther than arm's length), the location of the living room; or any combination of these. In some aspects, when location tracking is enabled, the decision component 134 may receive information that the audio instrument is in, for example, a restaurant or a performance venue.

[0068] The decision component 134 may select a volume for playback speech. For example, the decision component 134 may include a neural network component that outputs a volume level based on features of the user's speech, acoustic cues, non-acoustic cues, or a combination thereof. For example, the decision component 134 may determine that the listening environment is moderately noisy, the audio appliance is relatively close to the user, and the time of day is 10 a.m., and may select a modification to the playback volume without making other modifications to the playback speech. When the decision component 134 determines that a change in volume is required, the indicated change may be provided to the speech synthesizer 140 or the output DSP 150.

[0069] Decision making component 134 can determine the speech output mode to be used by speech synthesizer 140 according to the determined listening environment. Decision making component 134 can select the speech output mode 130 corresponding to the speech mode classified by the speech classifier. For example, selecting the speech output mode can include selecting one or more speech synthesis parameters 142 corresponding to the speech output mode. In some aspects, decision making component 134 can provide the selected speech synthesis parameters 142 to speech synthesizer 140, as will be further discussed below. In other aspects, decision making component 134 can alternatively provide the indication of the selected speech output mode 130 to speech synthesizer 140. In response to the indication, speech synthesizer 140 can select from a plurality of speech synthesis parameters according to the indication. In some aspects, when the determined speech output mode corresponds to normal speech, decision making component 134 can not provide any speech synthesis parameters 142 or speech output mode to speech synthesizer 140.

[0070] In some cases, decision component 134 may use one or more functions, mapping curves, lookup tables, or other relationships between their inputs and parameters to be selected for speech synthesis and / or to adapt playback volume and mode to the environment, such as by referencing Figure 4 Further discussed. In some cases, decision component 134 may include a neural network or other machine learning components, and may be trained to select specific parameters according to various training scenarios. Decision component 134 can, in response to receiving user feedback indicating that the output speech is incomprehensible, uncomfortable to listen to, or insufficient for the environment, adjust its training, such as by modifying the above-mentioned function, mapping curve, lookup table, etc. For example, a user may say "louder" or "I don't understand" after hearing a synthesized speech or other playback speech. Such utterances may prompt decision component 134 to select different parameters or adjust the value of the selected parameter to adjust the playback and / or mode of the speech, thereby improving the comprehensibility of the output. In some cases, feedback from users of different appliances may be aggregated, for example, on a remote server, and may be used to improve the operation of the decision component.

[0071] The appliance 100 may also be configured to perform speech recognition tasks on utterances (e.g., utterance 102) received via one or more onboard and / or communication-coupled microphones (e.g., microphones 104-1, 104-2, 104-3). Simultaneously or in parallel with the operation of the decision-making component, the input DSP 132 may be configured to transmit the prepared signal to an automatic speech recognition (ASR) system, e.g., an on-device ASR system or a remote automatic speech recognition (ASR) system 110, such as by using a connection to the Internet, such as to a cloud-based system. The appliance 100 may perform some speech recognition (SR) tasks locally while requesting other SR tasks from a remote ASR system. In some examples, the appliance 100 may perform all SR tasks locally without requesting any tasks from a remote ASR system.

[0072] In some cases, the appliance 100 or the SR system 110 may perform a task in response to receiving recognized speech that includes a spoken command or request. For example, the automatic speech recognition (ASR) component 112 may extract words and phrases from the received audio signal. The ASR component 112 may interpret and / or map the extracted words and phrases to a task or command. The response generation component 116 may perform the task or command, and may generate or retrieve text as a result of performing the task or command. For example, the user's utterance may be "What's the weather like in Dallas today?" The response generation component 116 may look up the weather forecast for Dallas, Texas on the current date, and may retrieve or generate a text statement such as "It will be sunny in Dallas today with a high of 96 degrees Fahrenheit."

[0073] The text from the response generation component 116 can be provided to the speech synthesizer 140. The speech synthesizer 140 can have a speech synthesis model 144 for generating normal mode speech from the text. The speech synthesizer 140 can generate an audio signal with a sound corresponding to the text. The speech synthesis model 144 can specify a value, a range of values, or a filter to be applied for one or more aspects of the synthesized speech in the audio signal. For example, a parameter can specify a pitch or a pitch range. Another parameter can specify the sound level or energy content of the audio signal. Another parameter can specify the speech rate. In some cases, the speech synthesizer 140 can have more than one model, such as a separate model for male and female voices, or a separate model for different accents (such as British English, Australian English and American English), or a separate model for different languages ​​(such as Dutch, German, Japanese). However, these different models can usually be designed to produce normal mode speech.

[0074] When the speech synthesizer 140 receives the speech synthesis parameters 142 from the decision component 134, the speech synthesizer 140 may synthesize speech using the received parameters 142 instead of the corresponding parameters in the speech synthesis model 144. In some cases, one or more of the received parameters 142 may be used in addition to the speech synthesis model 144, for example, if the speech synthesis model 144 does not specify a value or range of values ​​for a particular aspect of synthesized speech. The speech synthesis parameters may include values ​​for an aspect of speech, such as, for example, volume, pitch, rate, and pauses between words.

[0075] In aspects where the speech synthesizer 140 receives an indication of a speech output mode, the speech synthesizer 140 may select a speech synthesis parameter from among a plurality of speech synthesis parameters corresponding to the indication of the speech output mode.

[0076] The speech synthesis parameters may specify the use of specific filters. For example, a frequency shaping filter may be specified to raise the sound level of selected frequencies above the noise level. In some cases, a filter that modifies phase, a filter that modifies pitch, a filter that compresses phonemes, a filter that expands phonemes, a filter that normalizes power, or a combination thereof may be specified to improve the intelligibility of the synthesized speech.

[0077] The synthesized speech may be provided to an output component that includes an output digital signal processor (DSP) 150 configured to process the synthesized speech for output, and a speaker 106. In some aspects, the output component may include a speech synthesizer 140.

[0078] In many cases, the speech mode of the output synthesized speech may match the speech mode classification of the speech determined by the speech classifier 126, for example, when the user's speech is whispered, the output will be whispered speech, or when the user's speech exhibits a Lombard effect, the output will be Lombard effect speech. However, in some cases, the speech mode of the output synthesized speech may not match the speech mode classification of the speech. For example, if the user speaks in a normal mode in a noisy environment, the selected speech output mode 130 may be for Lombard effect speech rather than for normal speech.

[0079] IV. Example 2

[0080] Figure 2 An example of a second system 20 for adapting playback of a voice assistant device to an environment to preserve or improve the intelligibility of synthesized speech or other playback speech is shown. The system 20 may include an audio device 200 and may be similar in some aspects to Figure 1System 10. For example, system 20 may also use output DSP 150. However, embodiment 1 is to select from different speech synthesis parameters, and embodiment 2 is to select from different speech synthesis models. In system 20, speech synthesizer 240 may include multiple speech synthesis models, such as models 244-1, ... 244-n. The multiple speech synthesis models may include a speech synthesis model specific to whispered speech, a second speech synthesis model specific to Lombard effect speech, and a third speech synthesis model specific to normal speech. More, fewer or different speech synthesis models may be included. For example, there may be a single speech synthesis model that synthesizes all types of speech.

[0081] In system 20, appliance 200 may include input unit 220, which may include activation components (not shown) as described above and input DSP 232. Input DSP 232 may function similarly to input DSP 132, for example, by processing, filtering, or otherwise conditioning or preparing the collected representation of the ambient sound for use in a speech recognition task. The processed signal may be provided to remote ASR 210, which may include speech classifier 226. Speech classifier 226 may be similar to speech classifier 126 ( Figure 1 The speech classifier 226 may classify the speech before the ASR 112 performs speech recognition or concurrently with the operation of the ASR 112.

[0082] Decision component 234 can receive the result of speech classification directly from speech classifier 226 or from another component of ASR system 210. Decision component 234 can use speech classification, acoustic clues 136 and non-acoustic clues 138 to select speech output mode 230 according to speech classification, acoustic clues and non-acoustic clues. For example, selecting speech output mode 230 may include selecting a speech synthesis model 248 corresponding to the speech output mode, rather than a set of speech synthesis parameters. For example, when decision component 234 determines that the output synthesized speech should be whispered, decision component 234 can select a speech synthesis model specific to whispered speech. Decision component 234 can provide the indication of the selected speech model to speech synthesizer 240. Therefore, speech synthesizer can use the selected speech model when synthesizing speech.

[0083] The decision component 234 may also or alternatively select a playback volume for playing back speech (e.g., output speech). For example, the decision component 234 may include a neural network component that outputs a volume level based on features of the user's speech, acoustic cues, non-acoustic cues, or a combination thereof. As another example, the decision component 234 may use a mapping or lookup table to select a playback volume level based on one or more of features of the user's speech, acoustic cues, and non-acoustic cues. The selected playback volume level may be provided to the speech classifier 226 or the output DSP 150.

[0084] Although in Figure 2 2 is depicted as being on the ASR system 210, but in some aspects, the speech classifier 226 may instead be on the appliance 200, e.g., as Figure 1 shown.

[0085] V. Example 3

[0086] Figure 3 An example of a third system 30 for adapting playback of a voice assistant device to an environment to preserve or improve the intelligibility of synthesized speech or other playback speech is shown. The system 30 may include an audio device 300 and may be similar to Figure 1 and Figure 2 1 and 20, respectively. Embodiments 1 and 2 affect how speech is synthesized relative to the selected speech output mode, while in embodiment 3, the synthesized speech is modified according to the speech output mode after synthesis. For example, system 30 may also use input DSP 132, speech classifier 126, ASR system 110, and output DSP 150 as in system 10, or may use input DSP 232, speech classifier 226, ASR system 210, and output DSP 150 as in system 20.

[0087] The speech synthesizer 340 may synthesize speech without input from the decision component 334. In some cases, the speech synthesizer 340 may operate remotely from the audio instrument 300.

[0088] On the contrary, decision component 334 can select speech output mode 130. In one aspect, selecting speech output mode can include selecting one or more speech modification parameters according to speech mode classification, acoustic clues and non-acoustic clues. Decision component 234 can also or alternatively select the playback volume for playing back speech. The selected playback volume, speech modification parameters or both can be provided to speech modification component 342. Speech modification component 342 can be configured to receive synthesized speech from speech synthesizer 340, and modify the synthesized speech according to playback volume and / or selected one or more speech modification parameters. For example, speech modification parameters can make speech modification component 342 change the spectrum tilt of synthesized speech to correspond to Lombard effect speech, can remove pitch from synthesized speech to correspond to whispered speech, can only change the playback volume of normal speech output mode, or can change playback volume in addition to speech features.

[0089] In another aspect, selecting a speech output mode may include indicating the type (or classification) of speech to be output to speech modification component 342. Speech modification component 342 may then select speech modification parameters based on the indication of the type of speech to be output.

[0090] In one aspect, the speech modification component 342 can divide the synthesized speech into multiple frames. Each frame can be classified as voiced or unvoiced. The spectrum of each voiced frame can then be extracted, for example, by using a discrete Fourier transform. The spectrum in each voiced frame can then be modified using linear predictive coding (LPC) coefficients and received speech modification parameters. Each modified frame can then be rescaled to the same root mean square (RMS) sound level as the input synthesized speech. The signal can be reconstructed and input to a dynamic range compressor to compensate for the low sound level of the speech frame and to compensate for the user's hearing level.

[0091] The decision making component (e.g., decision making component 134, 234, and / or 334) may use one or more functions, mapping curves, lookup tables, or other relationships between their inputs and playback volume, parameters for selecting speech synthesis parameters, modifying parameters, or speech models. The values ​​of the various input variables may be inputs for one or more functions, or may be mapped to other values ​​in, for example, a lookup table. Figure 4 shows the corresponding v 1 、v 2 and v 3 The three input variables (variable 1, variable 2, and variable 3) represent the values ​​of the three variables. These three variables correspond to the point P in the three-dimensional space. env These variables may represent, for example, reverberation ratio, background noise, and signal-to-noise ratio. More, fewer, or other variables may be used in a space of higher, lower, or same dimensions.

[0092] The variable value can be input into the function F(v 1,v 2 ,…v n ), the function can operate on variables to obtain results. The result can correspond to speech classification, playback volume or both in some way. The modification to the synthesized speech can be selected corresponding to the speech classification. The result can correspond to the value mapped to a specific speech synthesis model, or to one or more parameters to be applied when synthesizing speech. In one example, the different speech modes that can be generated by the system can be represented by a set of intervals, wherein each corresponding speech mode has a separate interval, such as a whisper interval, a normal interval and a Lombard effect interval. Each corresponding interval can be associated with a numerical value of a different range, such as range 1, range 2 and range 3. Function F() can generate a value, and the value can be within an interval range in the interval range. The decision component selects a specific set of speech synthesis parameters, a specific speech synthesis model or a specific set of modification parameters according to the interval corresponding to the generated value.

[0093] The variables in function F() may be weighted relative to each other. In just one example, a variable reflecting the detected speech pattern of the user's speech may have a weight of 50%, while the remaining variables related to acoustic cues and non-acoustic cues may have a combined weight of 50%. This weighting may be used when matching the speech pattern of the output synthesized speech to the speech pattern classification of the speech has a higher priority than making the output speech more understandable in the listening environment, for example, making the output a whispered speech when the user's speech is whispered, or making the output speech a Lombard effect speech when the user's speech exhibits a Lombard effect.

[0094] In other cases, variables related to acoustic cues and non-acoustic cues may together have a weight that exceeds the speech mode variable. For example, this weight may be used when making the output speech understandable in the listening environment takes precedence over matching the speech mode of the user's utterance. For example, if the user speaks in a normal mode in a noisy environment, the output speech may be Lombard effect speech instead of normal speech.

[0095] In other cases, variables associated with acoustic cues may collectively be weighted more than variables associated with non-acoustic cues, and vice versa.

[0096] VI. Control Logic Components

[0097] See now Figure 5, describes an example of control logic. The illustrated control logic 500 may be executed, for example, by the audio device 100, such as by the input DSP 132, the speech classifier 126, the decision component 134, and the speech synthesizer 140, or by the audio device 200, such as by the input DSP 232, the speech classifier 226, the decision component 234, and the speech synthesizer 240. For the system 10 or the system 20, the illustrated control logic 500 causes the decision component 134, 234 of the corresponding system to affect how the speech is played back for the current environment of the audio device when synthesizing speech. Influencing how the speech is played back may include selecting a playback volume, selecting how the speech is synthesized, or both.

[0098] In block 502, the control logic 500 may receive an audio signal. The received audio signal may include an utterance, such as utterance 102, which in turn may include one or more words, including an utterance command or request. In some aspects, the received audio signal may include one or more utterances. The audio signal may be received by an input DSP (e.g., Figure 1 The input DSP 132) is received from the audio acquisition module and / or from the additional microphones 104-2, 104-3.

[0099] In block 504, the control logic component 500 may perform digital signal processing on the received audio signal. For example, an input DSP (e.g., Figure 2 The input DSP 232 of the audio signal may process, filter or otherwise prepare the audio signal for speech recognition tasks. For example, the input DSP may remove noise, perform echo cancellation, and remove reverberation from the audio signal.

[0100] In block 506, the control logic component 500 may classify the utterance according to the speech pattern. For example, a speech classifier (e.g., Figure 1 The speech classifier 126 of the speech classifier can analyze, extract, measure or otherwise determine one or more features of the speech to classify the speech mode of the speech. For example, the speech classifier can classify the speech mode as a whisper mode, a normal mode, or a Lombard effect mode. In one example, the speech classifier can compare, for example, the pitch of the speech to a pitch threshold or range, compare the spectral tilt of the speech to a spectral tilt threshold or range, and / or compare the energy content of the speech to an energy content threshold or range, or use a machine learning system.

[0101] In block 508, control logic 500 may provide the speech pattern classification to the decision making component. For example, a speech classifier (e.g., Figure 2The speech classifier 226 of the speech classifier 226 may transmit an indicator of the speech mode (e.g., at block 512, to be used by the control logic component 500 to select the speech synthesis mode). The indicator may be a number corresponding to the speech mode, or may be a text string corresponding to the speech mode.

[0102] In block 510, the control logic component 500 may identify a condition associated with the environment. For example, an input DSP (e.g., Figure 1 The input DSP 132 of the audio signal can extract, measure and / or analyze the audio signal to determine acoustic cues 136, such as background noise level, direct-to-reverberation sound energy ratio, signal-to-noise ratio, and echo coupling residual, and can provide the acoustic cues 136 to the decision making component (e.g., Figure 1 134). Additionally, the decision making component may request or retrieve non-acoustic cues 138 that identify non-acoustic conditions associated with the environment.

[0103] In block 512, the control logic component 500 may select a playback volume, a speech output mode from a plurality of speech output modes, or both. Figure 1 The decision component 134 of the embodiment may select the speech synthesis parameters 142 based on the conditions and speech mode associated with the environment (e.g., provided by block 508). Alternatively, the decision component 234 may provide an indication of the playback volume or the selected speech output mode. The decision component (e.g., Figure 2 The decision component 234 may be derived from a speech classifier (e.g., Figure 2 The speech classifier 226 of the embodiment receives the speech pattern and can select the playback volume, speech synthesis model 248 ( Figure 2 ) or both.

[0104] Simultaneously or in parallel with blocks 506 and 510, control logic 500 may also request automatic speech recognition of a spoken command or request in block 514. For example, the input DSP may request speech recognition of the processed audio signal from the ASR system 110 after some processing of the audio signal.

[0105] In block 516, control logic 500 may receive text generated by the ASR. For example, the text may correspond to an answer to a question, a confirmation that a task has been performed, a request for more information from a voice assistant, or other text to be communicated to a user as a result of the ASR operation.

[0106] In block 518, control logic 500 may output synthesized speech according to the text. For example, in system 10, speech synthesizer 140 may temporarily change one or more parameters of speech synthesis model 144 according to speech synthesis parameters 142 (or an indication of the received speech output mode), and synthesize the received text into speech according to the speech synthesis model. In system 20, speech synthesizer 240 may synthesize speech from the received text using speech synthesis model 244 indicated by decision component 234. Digital signal processor 150 may process synthesized speech for output through speaker 106.

[0107] In some aspects, if the decision component 134 or 234 determines that the output speech does not need to be modified, for example, if the speech classification of the utterance is normal and the environment is not noisy, the decision component may not output any speech synthesis parameters or the selected model at block 512. However, the decision component may modify the playback volume so that the speech is comfortable and not too loud or too quiet for the user's environment.

[0108] See now Figure 6 , describes another example of control logic. The illustrated control logic 600 may be implemented by the audio device 300 ( Figure 3 ), for example, by input unit 320, speech synthesizer 340, and speech modification component 342. The illustrated control logic 600 enables decision-making components of system 30 to influence how synthesized speech is scaled and / or modified for the current environment of the audio instrument.

[0109] The control logic 600 may be similar to Figure 5 602, 604, 606, 608, 610, 614, and 616 may be similar to blocks 502, 504, 506, 508, 510, 514, and 516, respectively. In block 612, control logic 600 may select one or more speech modification parameters based on the speech output mode. For example, decision component 334 may select one or more speech modification parameters based on the speech mode classification and the conditions identified in block 610. Alternatively, decision component 334 may select playback volume, select a speech output mode from a plurality of speech output modes, or both based on the speech mode classification, acoustic cues, non-acoustic cues, or a combination thereof.

[0110] In block 614, control logic 600 may simultaneously request speech recognition of spoken commands in the audio signal, which may be similar to block 514. For example, after performing some processing on the audio signal in block 604, input DSP 132 may request speech recognition on the processed audio signal from ASR system 110.

[0111] In block 616, the control logic 600 may receive text to be synthesized from the ASR system 110. For example, the text may correspond to an answer to a question, a confirmation that a task has been performed, a request for more information from a voice assistant, or other text to be communicated to a user as a result of an ASR operation.

[0112] In block 618, the control logic 600 may synthesize speech from the received text. For example, the speech synthesizer 340 may synthesize speech according to a speech synthesis model. The speech synthesis model may be a generic model that generates speech in a normal speech mode according to user settings that select, for example, gender, accent, and language for synthesized speech.

[0113] In box 620, control logic 600 may change synthesized speech according to voice modification parameters, playback volume, or both. For example, in order to produce the Lombard effect from normal mode synthesized speech, the voice modification parameters may cause the voice modification component 342 to increase the sound level or energy content at higher frequencies, and reduce the sound level or energy content at lower frequencies to reduce spectral tilt. The voice modification parameters may also cause the voice modification component 342 to slow down the speech rate of the synthesized speech, increase the pitch, and increase the sound level or energy content. For another example, in order to produce a whispered speech from normal mode synthesized speech, the voice modification parameters may cause the voice modification component 342 to remove the pitch and reduce the sound level or energy content. In some aspects, the voice modification component 342 may receive an indication of the selected speech output mode, and may select the playback volume, the voice modification parameters, or both themselves.

[0114] Then, in block 622, the control logic component 600 may output the modified synthesized speech. For example, the digital signal processor 150 may process the modified synthesized speech for output through the speaker 106.

[0115] In some aspects, if the decision component 334 determines that the output speech does not need to be modified, for example, if the speech classification of the utterance is normal and the environment is not noisy, then at box 612, the decision component 334 may output a playback volume modification but may not otherwise output any speech modification, and box 620 may be skipped.

[0116] VII. Effects of Position or Orientation on Comprehensibility

[0117] The location of the noise source relative to the user's location can inform the voice assistant device's decision about whether to modify the output speech or the degree to which the output speech is modified to improve the intelligibility of the output speech perceived by the user. For example, the voice assistant device (or other audio device) can be configured to distinguish the direction from which the device receives sound (e.g., noise) and the direction from which the device receives audible user input (e.g., user's speech). The device can also be configured to compare the relative arrival directions of the noise and the user input, and the comparison can inform whether the speech output mode can be modified from the normal speech output mode or the degree to which it should be modified in order to improve the intelligibility of the output speech perceived by the user.

[0118] For example, in a scenario where background noise (e.g., background noise from a coffee machine) arrives at the voice assistant device from one direction (sometimes referred to in the art as the "direction of arrival" or "DOA') and the user speaks from a different DOA relative to the voice assistant device, the playback sound level can be adjusted in a manner that is comfortable and intelligible to the user and not too loud. On the other hand, if the user speaks from the same or nearly the same DOA as the background noise, the playback sound level or other aspects of the output speech pattern can be adjusted, for example, the sound level can be increased to compensate for the background noise level and make the output speech relatively more intelligible. Although the previous sentence mentions sound level as an example of one feature of the synthesized output speech that can be adjusted or modified, other features of the synthesized output speech can be adjusted or modified according to the methods described herein in an attempt to improve the intelligibility of the output speech perceived by the user.

[0119] In other words, when the background noise arrives from a direction different from the direction from which the user's voice arrives, the voice assistant device may adjust the output voice mode (e.g., any one or more features of the output voice, including the output sound level) according to the selected first method. And, when the background noise arrives from the same or substantially the same direction as the user's voice arrives, the voice assistant device may adjust the output voice mode according to a different selected second method. For illustration, the voice assistant device may adjust the output voice mode according to the second method when the direction of arrival (DOA) of the background noise differs from the DOA of the user's voice by less than about 30 degrees (e.g., less than about 25 degrees), such as when the difference in DOA is less than about 20 degrees. For example, the output voice mode may be adjusted by increasing the output sound level, combining one or more other features of Lombard voice, and combining those adjustments or other adjustments of the output voice features.

[0120] As another example, when the difference in DOA is within a first range (e.g., between about 0 degrees and 10 degrees), the voice assistant device may select a first voice output mode. Alternatively, when the difference in DOA is within a second range (e.g., between about 10 degrees and about 20 degrees), the voice assistant device may select a second voice output mode. By way of another example, when the difference in DOA is within a third range (e.g., between about 20 degrees and about 30 degrees), the voice assistant device may select a third voice output mode. And, when the difference in DOA exceeds an upper threshold difference (e.g., greater than about 30 degrees), the voice assistant device may select a fourth voice output mode, such as a "normal" voice output mode.

[0121] As described above, a voice assistant device or other audio appliance can distinguish from which direction a sound (or sound component, such as background noise, user speech, or a combination thereof) is arriving at the device. For example, a voice assistant device may incorporate a microphone or microphone array that is configured to respond corresponding to the direction from which the sound is arriving at the device. Figure 7 , shows a microphone M 1 ,M 2 ,…,M 6 40. Although a circular array of six microphones is shown, any suitable number of microphones may be used in the microphone array and may be positioned in any suitable arrangement, for example, in an oval arrangement, a planar arrangement, a linear arrangement, or a three-dimensional arrangement, for example, in a cubic or ellipsoidal arrangement.

[0122] The acoustic response observed by each microphone in the microphone array can be used to determine the orientation of the microphone array relative to the incoming sound and the components of the incoming sound. Therefore, as long as the position of the microphone array relative to the audio device 40 is known, the orientation of the audio device and its component features (e.g., speaker array) can also be determined.

[0123] For ease of explanation, arrows 42a and 42b represent the first sound component and the second sound component arriving from different directions, respectively. Figure 5 In the orientation of the array 40 shown, the microphones M 1 The microphone M receives the least sound energy from the first sound component 42a, and 4 The most acoustic energy is received from the first sound component 42a (assuming that the sound component 42a arrives at the device 40 as a plane wave). In this example, the second sound component 42b arrives at the device 40 from a direction that is approximately orthogonal to the first sound component 42a. Assuming that the sound component 42b arrives at the device 40 as a plane wave, the point of minimum incident acoustic energy from the second sound component 42b is located at the microphone M. 2 and M 3and the point of maximum incident sound energy from the second sound component 42b is located at the microphone M 5 and M 6 between.

[0124] More generally, one or more components of incoming sound can be identified, characterized, or extracted from the sound observed by the microphone array. For example, voiced and unvoiced speech can be detected in the sound field and isolated from noise or other sound components. 1 ,M 2 ,…,M 6 To determine or estimate a measure of incident acoustic energy corresponding to each sound component. For example, the voice assistant device 10 may calculate the acoustic energy of each corresponding microphone M in the microphone array. 1 ,M 2 ,…,M 6 Similarly, the magnitude of each difference in frequency response can be calculated across a selected frequency band, for example, in a mid-range frequency band typical of human speech (e.g., voiced and unvoiced speech), with each difference corresponding to a frequency response extending around Figure 7 A corresponding microphone position among the microphone positions on the longitudinal axis in the page.

[0125] Because each external microphone is spatially independent relative to the other microphones, the calculated frequency response differences (or other measures of incident energy) associated with each microphone may differ from the calculated frequency response differences (or other measures of incident energy) associated with the other microphones.

[0126] Figure 7 The graphs 43a, 43b shown graphically depict the performance of each microphone M in the microphone array 40 for the first sound component 42a and the second sound component 42b, respectively. 1 ,M 2 ,…,M 6 As shown in graph 42a, the magnitude of the radiation impedance at microphone M 1 The minimum is at the microphone M 4 Similarly, graph 42b depicts the maximum value of microphone M 5 and M 6 The maximum radiation impedance between the microphone M 2 and M 3 The minimum radiation impedance between

[0127] Therefore, the aforementioned information can be used to estimate the direction of arrival of each incoming component of the sound. 1 ,M 2 ,…,M6 The position relative to the longitudinal axis may represent an angular coordinate in the r-θ plane defined by the circular microphone array 40 .

[0128] Furthermore, recognizing that the magnitude of incident acoustic energy of a sound component associated with or representative of human speech is maximized in the direction from which the sound component originates (e.g., the position of the speaker), the angular position of the maximum incident acoustic energy can be determined (e.g., based on Figure 7 ) to determine the orientation of the microphone array 40 relative to the speaker (and therefore the orientation of the corresponding voice assistant device). Using this direction estimate, the voice assistant device can affect the mode of device operation.

[0129] For example, the device may provide output speech in a selected manner, e.g., selecting an appropriate speech output mode as described herein to attempt to improve the intelligibility of the output speech perceived by the user, providing the output speech in a direction toward the speaker, or both.

[0130] Thus, the desired speech output mode or other operating mode can be automatically determined without interrupting the user, while still allowing the user to reposition the audio device and / or move throughout the listening environment. Such an audio device can avoid inconvenience to the user and improve the user's overall listening experience.

[0131] VIII. Computing Environment

[0132] Figure 7 An example of a suitable architecture for an audio appliance that can perform one or more tasks in response to commands spoken by a user or other information within an ambient sound field is shown. The audio appliance 700 can be a generalized representation of the audio appliances 10, 20, and / or 30. The audio appliance 700 includes an audio acquisition module 710, which can be an example of the AAM 122. The audio appliance 700 also includes various aspects of the computing environment that can enable the appliance to respond to the acoustic scene in a defined manner (e.g., as described below in conjunction with Figure 8 For example, the illustrated device 700 includes a processing unit 740 and a memory 750, the memory containing instructions that the processing unit can execute to cause the audio device to perform, for example, one or more aspects of the functions of collecting ambient sounds, extracting information from the collected sounds, and providing input unit 120 and / or speech synthesizer 140.

[0133] For example, such instructions may cause the audio instrument 700 to capture ambient sounds with the aid of the audio capture module 710 and transmit some or all of the captured audio signals to a remote speech recognition system (not shown) via the communication connection 760. The captured ambient sounds may include a user's spoken commands, such as, for example, "Hey, Siri, play the song [title]." By way of another example, the speech recognition system may extract such spoken commands from the audio signals and transmit a machine-readable command to the audio instrument 700 via the communication connection 760. The command may invoke an audio codec (not shown) and / or other instructions to cause the audio instrument to play the song through the speaker 770, as requested by the user.

[0134] Still reference Figure 7 , audio instruments typically include a microphone transducer to convert an incident acoustic signal into a corresponding electrical output. As used herein, the terms "microphone" and "microphone transducer" are used interchangeably and refer to an acoustic-electric transducer or sensor that converts an incident acoustic signal or sound into a corresponding electrical signal representing the incident acoustic signal. Typically, the electrical signal output by the microphone is an analog signal.

[0135] Although in Figure 7 A single microphone is depicted in , but the present disclosure contemplates the use of multiple microphones, such as microphones 104-1, 104-2, and 104-3. For example, multiple microphones may be used to obtain multiple different acoustic signals emanating from a given acoustic scene, and multiple versions may be processed independently and / or combined with one or more other versions and then further processed by the audio instrument 700.

[0136] like Figure 7 As shown, the audio acquisition module 710 may include a microphone transducer 720 and a signal conditioner 730 to filter or otherwise condition the acquired representation of the ambient sound. Some audio instruments have analog microphone transducers and preamplifiers to condition the signal from the microphone.

[0137] The output from the preamplifier or other conditioner may be low pass filtered before being sampled by the analog-to-digital converter (ADC), but the output does not have to be low pass filtered before sampling. In some cases, the ADC may use Sigma-Delta modulation and then use a decimation filter to downsample the digital signal.

[0138] For example, the analog output from the microphone transducer 720 may be low pass filtered and then sampled at the Nyquist frequency (i.e., twice the highest frequency component in the underlying signal to be sampled) or higher (e.g., having an upper frequency threshold corresponding to human hearing) relative to a selected passband to generate an acoustic data stream corresponding to the analog audio signal collected by the microphone 720. Assuming that the ambient sound of interest has an upper threshold frequency of approximately 20 kHz, a sampling frequency above approximately 40 kHz may accurately capture the acoustic scene of interest.

[0139] Figure 8 A generalized example of a suitable computing environment 800 is shown in which the described methods, aspects, techniques, and technologies related to, for example, maintaining the temperature of a logic component and / or a power unit below a threshold temperature may be implemented. The computing environment 800 is not intended to suggest any limitation on the scope of use or functionality of the techniques disclosed herein, as each technique may be implemented in a variety of general-purpose or special-purpose computing environments. For example, each disclosed technique may be implemented with other computer system configurations, including wearable devices and / or handheld devices (e.g., mobile communication devices, and more particularly, but not exclusively, the iPod Touch available from Apple Inc. (Cupertino, CA). / / HomePod TM Devices), multiprocessor systems, microprocessor-based or programmable consumer electronics, embedded platforms, network computers, minicomputers, mainframe computers, smart phones, tablet computers, data centers, audio appliances, etc. Each disclosed technology can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication connection or a network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0140] The computing environment 800 includes a memory 820 and at least one central processing unit 810. Figure 8 , this most basic configuration 830 is included within the dashed line. The central processing unit 810 executes computer executable instructions and can be a real or virtual processor. In a multi-processing system or in a multi-core central processing unit, multiple processing units execute computer executable instructions (e.g., threads) to increase processing speed, and therefore, multiple processors can run simultaneously, although the processing unit 810 is represented by a single functional block. The processing unit may include an application specific integrated circuit (ASIC), a general-purpose microprocessor, a field programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures arranged to process instructions.

[0141] The memory 820 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory 820 stores software 880a, which, when executed by the processor, may, for example, implement one or more of the techniques described herein.

[0142] The computing environment may have additional characteristic structures. For example, the computing environment 800 includes a storage device 840, one or more input devices 850, one or more output devices 860, and one or more communication connections 870. An interconnection mechanism (not shown) such as a bus, a controller, or a network interconnects the components of the computing environment 800. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing environment 800 and coordinates the activities of the components of the computing environment 800.

[0143] The storage device 840 may be removable or non-removable and may include selected forms of machine-readable media. Generally, machine-readable media include disks, magnetic tapes or cartridges, non-volatile solid-state memory, CD-ROMs, CD-RWs, DVDs, magnetic tapes, optical data storage devices and carrier waves, or any other machine-readable media that can be used to store information and can be accessed within the computing environment 800. The storage device 840 may store instructions for the software 880b that may implement the techniques described herein.

[0144] The storage device 840 may also be distributed over a network to store and execute software instructions in a distributed manner. In other aspects, some of these operations may be performed by specific hardware components that contain hardwired logic. Alternatively, those operations may be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0145] Input device 850 may be any one or more of: a touch input device, such as a keyboard, keypad, mouse, pen, touch screen, touchpad, or trackball; a voice input device, such as a microphone transducer, voice recognition software, and processor; a scanning device; or another device that provides input to computing environment 800. For audio, input device 850 may include a microphone or other transducer (e.g., a sound card or similar device that accepts audio input in analog or digital form), or a computer-readable medium reader that provides audio samples to computing environment 800.

[0146] Output device 860 may be any one or more of a display, a printer, a speaker transducer, a DVD writer, or another device that provides output from computing environment 800 .

[0147] The communication connection 870 enables communication with another computing entity through a communication medium (e.g., a connection network). The communication connection portion may include a transmitter and a receiver suitable for communicating through a local area network (LAN), a wide area network (WAN) connection, or both. LAN and WAN connections may be facilitated by wired or wireless connections. If the LAN or WAN connection is wireless, the communication connection portion may include one or more antennas or antenna arrays. The communication medium transmits information such as computer executable instructions, compressed graphics information, processed signal information (including processed audio signals) or other data in a modulated data signal. Examples of communication media for so-called wired connections include fiber optic cables and copper wires. Communication media for wireless communication may include electromagnetic radiation in one or more selected frequency bands.

[0148] Machine-readable media are any available media that can be accessed within computing environment 800. By way of example and not limitation, within computing environment 800, machine-readable media include memory 820, storage devices 840, communication media (not shown), and any combination thereof. Tangible machine-readable (or computer-readable) media do not include transitory signals.

[0149] As described above, some of the disclosed principles may be embodied in a tangible, non-transitory machine-readable medium (such as a microelectronic memory) having instructions stored thereon. The instructions may program one or more data processing components (collectively referred to herein as "processors") to perform the above-described processing operations, including estimation, calculation, computation, measurement, adjustment, sensing, measurement, filtering, addition, reduction, inversion, comparison, and decision making (such as by a decision making component). In other aspects, some of these operations (of a machine process) may be performed by specific electronic hardware components including hard-wired logic components (e.g., dedicated digital filter blocks). Alternatively, those operations may be performed by any combination of programmed data processing components and fixed hard-wired circuit components.

[0150] VIII. Other aspects

[0151] The above embodiments generally relate to apparatus, methods, and related systems for adjusting the playback of a voice assistant device, improving the user's experience, such as by making speech more understandable and adequate for each specific environment. More specifically, but not exclusively, the disclosed principles relate to systems, methods, and components for adjusting speech playback in a voice assistant device to improve the intelligibility of synthesized speech during or after synthesis based on the conditions of the listening environment and the user's speech pattern.

[0152] Nevertheless, the previous description is provided to enable those skilled in the art to make or use the disclosed principles. Aspects other than those described in detail above are contemplated based on the principles disclosed herein, and any accompanying changes in the configuration of the corresponding devices or changes in the order of the method actions described herein, without departing from the spirit or scope of the present disclosure. Various modifications to the examples described herein will be apparent to those skilled in the art.

[0153] For example, in some cases, aspects of the audio appliance may receive a voiced utterance from a user indicating that the output synthesized speech was not heard or understood, e.g., the user may say "say that again" or "I don't understand". When the user's utterance is processed by the ASR system and the voice assistant application, the ASR system or the voice assistant application may indicate to the decision component that the playback speech needs to be adjusted to improve intelligibility before repeating the output of the playback speech (including the synthesized speech). The decision component may adjust the playback volume, speech synthesis parameters, select a different speech synthesis model, and / or adjust speech modification parameters in response to the indication. Previously synthesized text may be resynthesized or may be modified using the adjusted parameters and then output.

[0154] As described above, one aspect of the present technology is to collect and use data obtained from specific and legitimate sources to improve the delivery of inspirational content or any other content that may be of interest to users. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, data or records related to the user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0155] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used to deliver targeted content that users may be more interested in based on their preferences. Therefore, the use of such personal information data enables users to have greater control over the content delivered. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0156] The present disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be highlighted and easily accessible to users, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legal use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0157] Regardless of the foregoing, the present disclosure also contemplates that users selectively block various aspects of the use or access to personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, such as with respect to an advertising delivery service, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for the service to participate in the collection of personal information data. In another example, a user may choose not to provide emotion-related data for a target content delivery service. As another example, a user may choose to limit the length of time that emotion-related data is retained, or completely block the development of basic emotional conditions. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then reminded again just before the personal information data is accessed by the application.

[0158] As described above, one aspect of the technology of the present invention is to collect and use data available from various sources to respond to user requests and commands with understandable synthetic speech. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or health level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0159] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used to provide requested information or perform tasks for users, and to convey requested information or information related to the performance of tasks in an understandable synthesized voice. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0160] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0161] Regardless of the foregoing, the present disclosure also contemplates that users selectively block various aspects of the use or access to personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a voice assistant application service, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for the service to participate in the collection of personal information data. As another example, a user may choose not to provide location-related data or acoustic clues for a speech intelligibility improvement service. As another example, a user may choose to limit the length of time that location-related data or acoustic clues are retained, or to completely prohibit the development of an acoustic profile for a certain location. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0162] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0163] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed aspects, the present disclosure also contemplates that various aspects may also be implemented without access to such personal information data. That is, various aspects of the present technology will not fail to function properly due to the lack of all or part of such personal information data. For example, requests may be responded to and tasks performed by inferring preferences based on non-personal information data or an absolute minimum measure of personal information (such as content requested by a device associated with a user, other non-personal information available to a voice assistant application service, or publicly available information).

[0164] Direction and other related references (e.g., up, down, top, bottom, left, right, backward, forward, etc.) may be used to help discuss the drawings and principles of this document, but are not intended to be limiting. For example, certain terms such as "upward", "downward", "upper", "lower", "horizontal", "vertical", "left", "right", etc. may be used. These terms are used to provide some clarity when dealing with relative relationships, especially with respect to the aspects shown, where applicable. However, such terms are not intended to imply absolute relationships, positions, and / or orientations. For example, with respect to an object, an "upper" surface can become a "lower" surface simply by flipping the object. Nevertheless, it is still the same surface and the object remains unchanged. As used herein, "and / or" means "and" or "or", as well as "and" and "or". In addition, for all purposes, all patents and non-patent literature cited herein are hereby incorporated by reference in their entirety.

[0165] Moreover, it should be understood by those skilled in the art that the exemplary aspects disclosed herein may be suitable for various configurations and / or uses without departing from the disclosed principles. Applying the principles disclosed herein, various principles related to improving the intelligibility of synthesized speech according to conditions associated with the listening environment, and related methods and systems for improving the intelligibility of synthesized speech according to conditions in the listening environment and user voice patterns can be provided. For example, the principles described above in conjunction with any specific example can be combined with the principles described in conjunction with another example described herein. Therefore, all structural and functional equivalents of the features and method actions of various aspects described throughout the present disclosure that are known or later known to those skilled in the art are intended to be covered by the principles described herein and the features and actions protected by the claims. Accordingly, neither the claims nor the present specific embodiments should be understood as restrictive, and after reading the present disclosure, those skilled in the art will recognize various principles related to improving the intelligibility of synthesized speech according to conditions associated with the listening environment, and related methods and systems that can be designed using the various concepts described herein.

[0166] Furthermore, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly recited in the claims. Claim features are not to be construed under 35 USC 112(f) unless the feature is expressly recited using the phrase "means for" or "step for."

[0167] The appended claims are not intended to be limited to the aspects shown herein, but rather are intended to have the full scope consistent with the language of the claims, wherein reference to features in the singular (such as by use of the article "a" or "an") is not intended to mean "one and only one," but rather "one or more," unless specifically noted. Furthermore, in view of the many possible aspects to which the disclosed principles may be applied, we reserve the right to claim any and all combinations of the features and techniques described herein as understood by one of ordinary skill in the art, including the right to claim, for example, all that is within the scope and spirit of the foregoing description, as well as in any claim issued at any time during the prosecution of this application or any application claiming the benefit or priority of this application, and more specifically but not exclusively the combinations literally and equivalently recited in the appended claims.

Claims

1. A method, comprising: Get audio signal; analyzing the audio signal to identify sounds; as well as In response to identifying the sound in the audio signal: classifying a speech pattern of the sound, the speech pattern indicating a first volume of the sound; determining acoustic cues from the audio signal; determining corresponding incoming wave directions for the sound and the acoustic cue; and A speech output mode is selected, wherein the speech output mode indicates a second volume based on the first volume, the acoustic clue, and the determined corresponding incoming wave direction. 2 . The method of claim 1 , wherein the acoustic cues include one or more additional sounds in the audio signal captured from a listening environment in which the audio signal was generated.

3. The method according to claim 1, wherein: The sound includes at least one word, and The speech pattern corresponds to the energy content of the at least one word.

4. The method according to claim 1, further comprising: determining a command from the sound; as well as In response to the command, synthesized speech is output in the selected speech output mode.

5. The method according to claim 1, further comprising: selecting a frequency of the audio signal; as well as The selected frequencies of the audio signal are filtered, wherein filtering the selected frequencies raises the sound level of the selected frequencies above the sound level of noise.

6. The method according to claim 1, further comprising: determining a problem from the sound; as well as A synthesized speech is output in the selected speech output mode as an answer to the question.

7. The method according to claim 1, wherein: Determining the corresponding incoming wave direction includes: Obtaining the first incoming wave direction of the sound; and obtaining a second wave direction of the acoustic clue, where the second wave direction is different from the first wave direction, and The speech output mode is further selected based on a difference between the first arrival direction and the second arrival direction.

8. The method according to claim 7, wherein: Acquiring the first incoming wave direction includes acquiring the audio signal from a first microphone, and Acquiring the second incoming wave direction includes acquiring the acoustic clue from a second microphone.

9. The method according to claim 1, further comprising: Identify non-acoustic cues including room layout; as well as The speech output mode is further selected based on the non-acoustic cues.

10. The method according to claim 1, further comprising: determining a non-acoustic cue selected from at least one of: time of day, and a determination that an appliance generating the selected speech output pattern is located in a vehicle; and The speech output mode is selected further based on the non-acoustic cues.

11. An apparatus comprising: one or more microphones; a memory configured to store instructions; as well as a controller configured to execute the instructions stored on the memory, the instructions causing the controller to: acquiring an audio signal from the one or more microphones; analyzing the audio signal to identify sounds; as well as In response to identifying the sound in the audio signal: classifying a speech pattern of the sound, the speech pattern indicating a first volume of the sound; determining acoustic cues from the audio signal; determining corresponding incoming wave directions for the sound and the acoustic cue; and A speech output mode is selected, wherein the speech output mode indicates a second volume based on the first volume, the acoustic clue, and the determined corresponding incoming wave direction.

12. The apparatus according to claim 11, wherein the controller is further configured to: determining a speech rate of the audio signal; and The selected speech output mode is further selected based on the speaking rate.

13. The apparatus according to claim 11, wherein the controller is further configured to: Using a first microphone among the one or more microphones to obtain a first direction of arrival of the sound; and using a second microphone of the one or more microphones to obtain a second wave direction of the acoustic clue, wherein the second wave direction is different from the first wave direction, and The corresponding wave arrival direction is determined based on the first wave arrival direction and the second wave arrival direction.

14. The apparatus of claim 11, wherein the controller is further configured to: using the acoustic cues to determine the distance between the appliance and the user; and The speech output mode is selected further based on the distance.

15. The apparatus of claim 11, wherein the controller is further configured to: selecting a frequency of the audio signal; and The selected frequencies of the audio signal are filtered, wherein filtering the selected frequencies raises the sound level of the selected frequencies above the sound level of noise.

16. A non-transitory computer readable medium comprising computer executable instructions, which, when executed, causes a computing environment in a device to implement the following steps: Get audio signal; analyzing the audio signal to identify sounds; as well as In response to identifying the sound in the audio signal: classifying a speech pattern of the sound, the speech pattern indicating a first volume of the sound; determining acoustic cues from the audio signal; Obtaining the first incoming wave direction of the sound; obtaining a second wave direction of the acoustic clue, wherein the second wave direction is different from the first wave direction, and A speech output mode is selected, wherein the speech output mode indicates a second volume based on the first volume, the acoustic cue, and a difference between the first arrival direction and the second arrival direction.

17. The non-transitory computer-readable medium of claim 16, wherein the acoustic cues include one or more additional sounds in the audio signal captured from a listening environment in which the audio signal was generated.

18. The non-transitory computer readable medium of claim 16, wherein the steps further comprise: determining a command from the sound; as well as In response to the command, synthesized speech is output in the selected speech output mode.

19. The non-transitory computer readable medium of claim 16, wherein the steps further comprise: Identify non-acoustic cues selected from at least one time of day; as well as The speech output mode is selected further based on the non-acoustic cues.

20. The non-transitory computer readable medium of claim 16, wherein the steps further comprise: determining a speech rate of the audio signal; as well as The selected speech output mode is further selected based on the speaking rate.

Images