Voice Processing System

Abstract

A system may include first and second speech processing systems. The first speech processing system may process received audio data and determine that a command represented therein is associated with a second speech processing system. The first speech processing system may send command data to the second speech processing system and receive response data in return. The first speech processing system may then process the response data to determine second response data including an indication of the second speech processing system and generate an audio output corresponding to the second response data.

Description

[0001] Cross-references to related application data

[0002] This application claims the benefit of priority to U.S. Patent Application No. 16 / 781,773, filed on February 4, 2020 by Timothy Whalin et al. and entitled “SPEECH-PROCESSING SYSTEM.” Background Art

[0003] Speech processing systems allow users to control computing devices using their voice. These systems recognize words spoken by a user based on the properties of received audio data representing the user's voice. Automatic speech recognition (ASR) processing combined with natural language understanding (NLU) processing allows speech processing systems to determine the text corresponding to the speech and understand the intent expressed in the text. ASR processing and NLU processing can be combined with text-to-speech (TTS) processing, which can be used to generate synthesized speech in response to human speech. Speech processing can be used by computers, handheld devices, smart speakers, and other devices to improve human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0005] Figure 1 A system configured to process user input using a speech processing system according to an embodiment of the present disclosure is shown.

[0006] Figure 2A A user device and a remote system configured for speech processing according to an embodiment of the present disclosure are shown.

[0007] Figure 2B A user device configured for speech processing according to an embodiment of the present disclosure is shown.

[0008] Figure 3A and Figure 3B A speech processing system according to an embodiment of the present disclosure is shown.

[0009] Figure 4A , Figure 4B and Figure 4C A method of using a speech processing system according to an embodiment of the present disclosure is shown.

[0010] Figure 5 A natural language processing system according to an embodiment of the present disclosure is shown.

[0011] Figure 6 Components of a natural language processing system according to an embodiment of the present disclosure are shown.

[0012] Figure 7 A text-to-speech processing component according to an embodiment of the present disclosure is shown.

[0013] Figure 8 A user identification component according to an embodiment of the present disclosure is shown.

[0014] Fig. 9 Example components of a user device according to an embodiment of the present disclosure are shown.

[0015] Fig.10 Example components of a remote system according to an embodiment of the present disclosure are shown.

[0016] Fig.11 An example of a computer network for a speech processing system is shown. DETAILED DESCRIPTION

[0017] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that is concerned with converting audio data representing speech into text data representing the speech. Natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics that is concerned with enabling computers to derive meaning from text data. Text-to-speech (TTS) is a field of computer science, artificial intelligence, and linguistics that is concerned with enabling computers to convert representations of text into audio representing synthesized speech. ASR, NLU, and / or TTS may be used together as part of a natural understanding system, which may also be referred to as a speech processing system.

[0018] The voice-controlled user device and / or remote system may be configured to receive spoken user input and detect wake-up words and / or other text in the user input using ASR; determine commands in the user input using NLU; and provide responses to commands using TTS and / or other user interfaces. The user can therefore control a voice-controlled user device, another user device, and / or a remote system by voice. In some embodiments, in response to the user device detecting the wake-up word, the user device may send audio data representing the user input to the remote system for further processing. The remote system may further process the audio data to verify that it includes a representation of the wake-up word and / or determine a command and / or response. The user device may then receive output audio, video, or other data associated with the response from the remote device and / or other data required to perform an action associated with the response.

[0019] The user device and / or remote system of voice control can use two or more speech processing systems to process user input, and each speech processing system can have its own ASR component, NLU component and / or TTS component. Each speech processing system can be associated with commands and / or responses to those commands unique to the speech processing system. For example, a user may prefer a first speech processing system for online shopping, and may prefer a second speech processing system for navigation. Therefore, a user may wish to direct online shopping commands to the first speech processing system and wish to direct navigation commands to the second speech processing system. The first speech processing system may not be able to complete some or all commands related to the second speech processing system, and vice versa. However, both speech processing systems may be able to execute other commands.

[0020] In addition to being associated with its own (possibly unique) interpretation of commands, each speech processing system may also provide its own (possibly unique) responses, and / or be otherwise considered to have a particular "flavor." For example, such a flavor may include a particular (albeit artificial) speaking voice and / or style, such as "newscaster" or "celebrity." A speech processing system may create a synthesized speech (e.g., using a TTS component) that includes a phonetic representation of various aspects of the flavor. Thus, a user may associate a speech processing system with a flavor, and thereby identify the synthesized speech as being associated with a particular speech processing system by identifying the flavor.

[0021] Each speech processing system may also be associated with a different wake-up word. For example, a first speech processing system may be associated with the wake-up word "Alexa". The user device and / or remote system may be configured to use a specific speech processing system when its associated wake-up word is detected. The user device and / or remote system may recognize more than one wake-up word; different wake-up words may be associated with different speech processing systems. For example, the user device and / or remote system may be configured to perform an action, such as lowering the window of a car, in response to a voice from a user including "SmartCar, roll down my window". In this example and throughout this disclosure, the wake-up word "SmartCar" is used to represent a wake-up word corresponding to a second speech processing system associated with a car; this second speech processing system may be able to perform car-specific actions that the first speech processing system cannot perform, such as raising / lowering windows, adjusting car seats, etc. However, the present disclosure is not limited to this wake-up word (or only the "Alexa" wake-up word), nor is it limited to cars.

[0022] However, in some cases, the user input includes a first wake-up word associated with a first speech processing system, but the command represented in the user input may only be executed by and / or may be more preferably executed by a second speech processing system. However, the user may wish to execute the command even if it was originally intended for the first speech processing system. For example, if the remote system includes ten different speech processing systems, each with its own unique wake-up word and commands, it may be easier for the system to determine the best wake-up word for a particular command rather than always relying on the user to remember which wake-up word is associated with which commands.

[0023] The first speech processing system may thus recognize that the second speech processing system is associated with the command and therefore cause further processing of the user input using the second speech processing system. The first and / or second speech processing system may cause the output of synthesized speech to notify the user of this change in processing so that the user is not surprised or confused by the change. For example, if the user says, "Alexa, roll down the window," the first speech processing system may cause the output of synthesized speech representing the following phrase: "I can't help you with this problem, so it's now handled by SmartCar [e.g., the name of a different artificial feature]." The second speech processing system behind the SmartCar artificial feature may then determine an appropriate response.

[0024] However, the user may sometimes not wish to hear such an indication of a processing change and / or may sometimes not wish to hear synthesized speech corresponding to the characteristics of the second speech processing system. For example, this may result in a delay in interaction, and in some cases, for various reasons, the user may prefer the characteristics of the first speech processing system. Thus, in at least some cases, making the response sound like it comes from one characteristic may be more preferred regardless of which speech processing system ultimately processes the speech. The user may similarly prefer a concise response and may not wish to hear a response that includes the attributes of the second speech processing system.

[0025] Therefore, in some cases, the first speech processing system can first determine that it cannot process commands and / or perform associated actions and / or the second speech processing system can better process commands and / or perform associated actions. Then, the first speech processing system can identify the second speech processing system, communicate with it so that it executes commands, and output the audio corresponding to the characteristics of the first speech processing system. In some embodiments, the first speech processing system can also output the indication of the second speech processing system. The indication can be, for example, a name, sound and / or voice style associated with the second speech processing system. In other embodiments, the first speech processing system can hand over the further processing of audio data to the second speech processing system.

[0026] Figure 1A system configured to determine a response to a command represented by audio data according to the present disclosure is shown. Although the figures and discussion of the present disclosure illustrate certain operating steps of the system in a particular order, the steps described may be performed in a different order (and certain steps may be removed or added) without departing from the intent of the present disclosure. In various embodiments, a user device 110, such as a smart speaker, vehicle, or other such voice-controlled device, communicates with a remote system 120 using a network 199. Although Figure 1 A smart speaker is shown as the user device 110, but the present disclosure is not limited in this regard, and the systems and methods described herein may be implemented using other user devices 110 such as a smartphone, tablet computer, personal computer, or other device.

[0027] The remote system 120 receives (130) input data representing a command. For example, the user device 110 may determine that the input audio 11 includes a representation of a wake word and send corresponding input audio data 211 to the remote system 120 via the network 199. To make this determination, the user device 110 may capture audio using one or more microphones and create corresponding audio data, which may be a time-domain digital representation of the audio amplitude and may be represented as a series of samples. The user device 110 may process this time-domain audio data to create processed audio data; for example, this processing may include performing one or more Fourier transforms, such as a fast Fourier transform (FFT), and / or grouping samples of the audio data into audio data frames. The user device 110 may include a wake word detector 220 (as described below with respect to Figure 2A and Figure 2B ), the wake-up word detector processes frames of audio data to determine whether a wake-up word is represented therein. The input audio data 211 may include representations of the wake-up word and / or representations of other words in addition to commands. The user device 110 may send additional data to the remote system 120, such as the time when the wake-up word was detected and / or an indication of which wake-up word was detected.

[0028] The remote system 120 may use the first natural language system to determine (132) that the command corresponds to the second natural language system. The first natural language system may include (e.g., reference Figure 2A and Figure 2BThe first natural language system may further include an ASR component 250 that processes the input audio data 211 to determine corresponding text data representing the command. The first natural language system may further include an NLU component 260 that processes the text data to determine the domain of the command, the intent of the command, and / or one or more of the one or more slots represented in the command. The natural language system identification component may then process the output of the NLU component 260 to determine which of the multiple natural language systems corresponds to (or most closely corresponds to) the command. The natural language system identification may include, for example, a list of natural language systems and associated domains and / or intents for each natural language system, and thus the natural language system may be identified by matching the domains and / or intents determined by the NLU component 260 with entries in the list.

[0029] The first natural language system may then send (134) command data corresponding to the command to the second natural language system. The command data may be input data, determined text data, and / or NLU data determined by NLU component 260. The second natural language system may then determine result data representing a response to the command based on the command data. The result data may be audio data representing synthesized speech; as described above, such synthesized speech may exhibit characteristics of the second natural language system. The result data may alternatively or additionally include text data, video data, and / or image data.

[0030] After the second natural language system sends the result data, the first natural language system receives (136) the result data. The result data may include an indication of the second natural language system; the indication may be the name of the second natural language system and / or text and / or audio data corresponding to characteristics of the second natural language system. The first natural language system may then use the result data to determine (138) output data; the output data 212 may include an indication of the first natural language system. The indication may similarly be the name of the first natural language system and / or text and / or audio data corresponding to characteristics of the first natural language system.

[0031] As described in more detail herein, the first natural language system may also determine (140) an attribute of the second natural language system during certain circumstances, which may include an output of an indication of the second speech processing system. In other circumstances, the first natural language system may also hand over processing of the audio data to the second natural language system. The term "handover" as used herein refers to the process by which the first natural language system sends a corresponding signal to the second natural language system; the first natural language system then stops processing the audio data, and the second natural language system begins processing the audio data.

[0032] The first speech processing system may then cause (142) an output of audio corresponding to the output data 12. Thus, although the command is ultimately executed by the second natural language system, the response to the user reflects the characteristics of the first natural language system.

[0033] The system can be used as Figure 2A and / or Figure 2B The various components described in the present invention may be located on the same or different physical devices. For example, Figure 2A As shown, some components may be provided on the user device 110, while other components may be provided on the remote system 120; however, Figure 2B As shown, the components may be provided on the user device 110. Communication between the various components may be performed directly (via, for example, a bus connection) or across the network 199. As described in more detail below, the user device 110 may include a wake-up word detector 220 for detecting one or more wake-up words, a voice activity detector 222 for detecting speech, and / or one or more applications 224 for providing output and / or changing the state of the user device 110 (e.g., emitting light). As shown, the wake-up word detector 220 is provided on the user device 110, while another speech processing component (e.g., NLU component 260) is provided on the remote system 120. However, the present disclosure is not limited to such a component arrangement, and any other component arrangement is within the embodiments of the present disclosure. For example, some or all speech processing may be performed by the user device 110, and therefore the user device 110 may not send any audio data 211 to the remote system 120.

[0034] An audio capture component (e.g., a microphone or microphone array of a user device 110) captures input audio 11 and creates corresponding input audio data 211. The wake-up word detector 220 of the user device 110 may process the input audio data 211 to determine whether speech is represented therein. The user device 110 may use various techniques to determine whether the input audio data 211 includes speech. In some examples, the voice activity detector 222 of the user device 110 may apply voice activity detection (VAD) techniques. Such VAD techniques may determine whether there is speech in the audio data based on various quantitative aspects of the input audio data 211, such as the spectral slope between one or more frames of the audio data; the energy level of the audio data in one or more spectral bands; the signal-to-noise ratio of the audio data in one or more spectral bands; or other quantitative aspects. In other examples, the user device 110 may include a classifier configured to distinguish speech from background noise. The classifier may be implemented by techniques such as linear classifiers, support vector machines, and decision trees. In other examples, user device 110 may apply a Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) technique to compare the audio data with one or more acoustic models in a storage device, which may include models corresponding to speech, noise (e.g., ambient noise or background noise), or silence. Still other techniques may also be used to determine whether speech is present in the audio data.

[0035] The wake word detector 220 may determine that the input audio data 211 contains a representation of the wake word (as described in more detail below); thereafter, the user device 110 may send the input audio data 211 to the system 120. As described above, example wake words are "Alexa" or "SmartCar". The user device 110 may alternatively or additionally send audio data to the system 120 when the input detector detects an input—such as a key press, a button press, or a touch screen touch. An example button is a "push to talk" button. In either case, the user device 110 sends the input audio data 211 to the server 120.

[0036] The wake-up word detector 220 may compare the audio data with a stored model or data to detect the wake-up word. One method for wake-up word detection applies a general large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, wherein a wake-up word search is performed in the resulting grid or confusion network. LVCSR decoding may require relatively high computing resources. Another method for wake-up word detection constructs an HMM for each wake-up word and non-wake-up word speech signal respectively. Non-wake-up word speech includes other spoken words, background noise, etc. One or more HMMs may be constructed to model the non-wake-up word speech characteristics, which are referred to as filler models. Viterbi decoding is used to search for the best path in the decoding graph, and the decoded output is further processed to make a decision about the presence of the wake-up word. This method can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake-up word detector 220 may be built directly on a deep neural network (DNN) / recursive neural network (RNN) structure without involving an HMM. This architecture can estimate the posterior of the wake-up word with context information by stacking frames within a context window of a DNN or using an RNN. Subsequent a posteriori threshold adjustments or smoothing are applied to the decision.Other techniques for wake-up word detection, such as those known in the art, may also be used.

[0037] Once the wake word detector 220 detects the wake word and / or the input detector detects the input, the user device 110 may transmit the audio data 211 to the remote system 120. The input audio data 211 may include data corresponding to the wake word; in other embodiments, the portion of the audio corresponding to the wake word is removed by the user device 110 before the input audio data 211 is sent to the remote system 120. For example, in the case of touch input detection, the input audio data 211 may not include the wake word.

[0038] In various embodiments, wake-up word detector 220 may use one of a plurality of wake-up word detection models. Each model may be trained to detect a different wake-up word. In some embodiments, a single model may detect multiple wake-up words. Each wake-up word may be associated with a different speech processing system. Upon detecting a particular wake-up word, user device 110 may send audio data 211 (and / or an indication that a wake-up word was detected) to its corresponding speech processing system.

[0039] In various embodiments, the wake word detection model of the wake word detector 220 is implemented to detect wake words spoken with different accents corresponding to different countries, regions, or other areas. For example, the wake word detection model may be implemented to detect whether the wake word "Alexa" is spoken with an Indian accent, a Scottish accent, or an Australian accent. The wake word detection model may also be implemented to detect other wake words in other languages; these other languages ​​may have similar accent variations that the wake word detection model may similarly be implemented to detect.

[0040] The wake-up word detector 220 may determine a similarity score for the candidate wake-up word based on the similarity of the candidate wake-up word to the stored wake-up word; if the similarity score is above the wake-up word detection threshold, the wake-up word detector 220 determines that the wake-up word is present in the audio data, and if the similarity score is below the wake-up word detection threshold, the wake-up word detector 220 determines that the wake-up word is not present in the audio data. For example, if the candidate wake-up word is very close to the stored wake-up word, the wake-up word detector 220 may determine a similarity score of 100; if the candidate wake-up word does not match the stored wake-up word at all, the wake-up word detector 220 may determine a similarity score of 0. If the wake-up word detector 220 determines that the candidate wake-up word partially matches the stored wake-up word, an intermediate similarity score may be determined, such as 75 or 85. Although the disclosure herein describes similarity scores of 0 to 100, with zero being the least similar and 100 being the most similar. However, the present disclosure is not limited to any particular range of values ​​for the similarity score, and any system or method that determines the similarity between a candidate wake word represented in captured audio data and a stored representation of the wake word is within the scope of the present disclosure.

[0041] The user device 110 may also use different wake-up words for different skills within the same speech processing system. Thus, the user device 110, using the techniques described herein, may process incoming audio to determine a first confidence that the detected wake-up word is a first wake-up word associated with a first speech processing system (which may be a first speech processing component (e.g., a skill, etc.) within the first speech processing system), and a second confidence that the detected wake-up word is a second wake-up word associated with a second speech processing system (which may be a second speech processing system or a second different component (e.g., a skill, etc.) within the second speech processing system). Different systems may be associated with different ASR processing, different NLU processing, different TTS processing, different domains / intents, or other differences.

[0042] After being received by the system 120 and / or after being determined by the user device 110, the input audio data 211 can be sent to the coordinator component 240. The coordinator component 240 may include memory and logic that enables the coordinator component to transmit various data segments and various forms of data to various components of the system and perform other operations as described herein. The coordinator component 240 can be or include one or more of the speech processing system manager and / or speech processing system 292, which can be used to determine which (if any) of the ASR 250, NLU 260, and / or TTS 280 components should receive and / or process the audio data 211. In some embodiments, the coordinator component 240 includes one or more ASR components 250, NLU components 260, TTS components 280, and / or other processing components, and processes the input audio data 211 and / or other data before sending it to one or more speech processing components 292 for further processing. In other embodiments, the coordinator component 240 sends the input audio data 211 to one or more speech processing components 292 for processing.

[0043] As described above, the coordinator component 240 (and / or one or more speech processing systems 292) can determine that the input audio data 211 is associated with the first speech processing system 292a (e.g., it contains a representation of a wake-up word associated with the first speech processing system 292a, or it does not contain any representation of the wake-up word and the first speech processing system 292a is selected by default). For example, the wake-up word detector 220 can detect the presence of a particular wake-up word and send an indication of the wake-up word to the speech processing system manager. Alternatively or in addition, the ASR component 250 can determine text data corresponding to the audio data 211 and send the text data to the speech processing system manager, which can then determine that the wake-up word is represented in the text.

[0044] The speech processing system identification component 296 can be used to determine that, although the input audio data 211 is associated with a first speech processing system, the command represented in the input audio data 211 corresponds to a second speech processing system. The speech processing system identification component 296 can include, for example, a list of speech processing systems and associated domains, intents, and / or entities in a computer memory. The speech processing system identification component 296 can receive the determined domains, intents, and / or entities from the NLU component 260 and select a matching speech processing system from the list. The speech processing system identification component 296 can also or alternatively include a training model, such as a classifier, which processes the audio data 211 (and / or text data determined therefrom) to output a corresponding speech processing system.

[0045] As referenced below Figure 4BAs explained, the user device 110 and / or the remote system 120 may also include an attribute determination component 298 for determining whether to output an attribute (e.g., an indication) of the second speech processing system. In other embodiments, the coordinator component 240 may hand over the processing of the audio data 211 from the first speech processing system 292a to the second speech processing system 292b.

[0046] The first speech processing system 292a may control, have access to, or otherwise be associated with a first set of skills 290, applications, and / or voice styles, and the second speech processing system 292b may control, have access to, or otherwise be associated with a second set of skills 290, applications, and / or voice styles. The first and second sets of skills 290 may include general skills as well as skills unique to each speech processing system 292. In some embodiments, the first speech processing system 292a is a general speech processing system and may provide such skills and applications as weather forecasts, restaurant reservations, shopping services, and Internet searches; the second speech processing system 292b is a vehicle-specific speech processing system and may provide such skills and applications as changing the state of the user device 110 (e.g., raising / lowering windows, setting thermostats, and / or adjusting seats) and / or providing diagnostic information. Some applications and skills may be common to both speech processing systems 292 (e.g., playing music or providing navigation information). However, any number of speech processing systems 292 with any type of application or skill are within the scope of the present disclosure.

[0047] In some embodiments, the speech processing system manager uses one or more components in the speech processing system 292 (e.g., the ASR component 250 and the NLU component 260) to direct the processing of the input data 311. In other embodiments, the speech processing system manager includes an NLU component for NLU processing; this NLU can be used to process all audio data, while other NLU components 260 in the speech processing system 292 can be used to process different audio data (and can be selected based on the wake-up words detected in the audio data). As explained in more detail below, the speech processing system manager and / or the speech processing system 292 can perform ASR and / or NLU processing to determine the domain, intent, and / or meaning corresponding to the audio data 211. The speech processing system manager and / or the speech processing system 292 can alternatively or additionally identify keywords in the input audio data 211 and identify the domain, intent, and / or meaning corresponding to the keywords (e.g., entity resolution). The speech processing system manager and / or the speech processing system 292 can further identify the user associated with the utterance in the input audio data 211 and identify the domain, intent, and / or meaning corresponding to the user. The speech processing system manager and / or speech processing system 292 can further identify the domain, intent and / or meaning associated with the input audio data 211 based on information in a user profile associated with the user (e.g., usage history information), the location of the user device 110, the time of day, week, month or year, and / or temperature information.

[0048] The speech processing system manager and / or the speech processing system 292 may compare the determined domains, intents, and / or meanings with a list of corresponding applications or skills associated with each speech processing system 292. The comparison may include determining the number of full or partial matches of the domains, intents, and / or meanings present in each list. The speech processing system manager and / or the speech processing system 292 may determine a score for each speech processing system 292 corresponding to the ability of each speech processing system 292 to respond to the command represented in the input audio data 211. If it is determined that the domain, intent, and / or meaning is associated with the first speech processing system 292 but not the second speech processing system 292, the speech processing system manager may award the first speech processing system 292 a higher score than the second speech processing system 292. If it is determined that the domain, intent, and / or meaning is associated with both the first speech processing system 292 and the second speech processing system 292, the speech processing system manager may determine the ranking based on other data such as user identification, user profile data, location, or other information.

[0049] In some embodiments, the coordinator 240 and / or the speech processing system manager communicates with the speech processing system 292 using an application programming interface (API). The API can be used to send data, commands, or other information to and / or receive data, commands, or other information from the speech processing system 292. For example, the coordinator 240 can send input audio data 211 to a speech processing system selected by the speech processing system manager via the API, and can receive commands and / or data in response to the audio data 211 from the selected speech processing system 292.

[0050] In some embodiments, as described above, the speech processing system manager includes processing components, such as ASR and / or NLU components, that can be used to select a speech processing system 292. Alternatively or in addition, in other embodiments, the speech processing system manager communicates with a specific speech processing system 292, such as via an API, to cause the speech processing system 292 to perform processing, and in response receives data corresponding to the processing and / or the selected speech processing system 292. Thus, the speech processing system manager may include one or more application programming interfaces (APIs) for communicating with a specific speech processing system 292, a configuration manager for determining properties of the user device 110, and / or an event processor for processing events received from the user device 110 and / or the speech processing system 292, but may not include an ASR processor or an NLU processor, which may be included in the specific speech processing system 292 instead.

[0051] Each speech processing system 292 may include an ASR component 250 that can transcribe the input audio data 211 into text data. The text data output by the ASR component 250 can represent one or more (e.g., in the form of an N-best list) ASR hypotheses that represent the speech represented in the input audio data 211. The ASR component 250 interprets the speech in the input audio data 211 based on the similarity between the audio data 211 and a pre-established language model. For example, the ASR component 250 can compare the input audio data 211 with a model of sound (e.g., an acoustic unit, such as a phoneme, a signal, a phoneme, etc.) and a sound sequence to identify words that match the sound sequence of the speech represented in the input audio data 211. In some embodiments, the ASR component 250 sends the text data generated thereby to the NLU component 260 via the coordinator component 240. The text data sent from the ASR component 250 to the NLU component 260 may include a single highest-scoring ASR hypothesis, or may include an N-best list including a plurality of highest-scoring ASR hypotheses. The N-best list may additionally include a respective score associated with each ASR hypothesis represented therein.

[0052] Each speech processing system 292 may also include Figure 5 and Figure 6 , which attempts to semantically interpret phrases or sentences represented in the text data input therein by determining one or more meanings associated with the phrases or sentences represented in the text data. The NLU component 260 can determine an intent representing an action that the user desires to perform and can determine information that allows a device (e.g., user device 110, system 120, skill component 290, skill system 225, etc.) to perform the intent. For example, if the text data corresponds to "play Africa by Toto", the NLU component 260 can determine the intent of the system to output music and can identify "Toto" as an artist and "Africa" ​​as a song. For another example, if the text data corresponds to "how's the weather", the NLU component 260 can determine the intent of the system to output weather information associated with the geographic location of the user device 110. In another example, if the text data corresponds to "turn off the lights", the NLU component 260 can determine the intent of the system to turn off the lights associated with the user device 110 or user 5.

[0053] The NLU result data may be sent from the NLU component 260 (which may include tagged text data, intent indicators, etc.) to the skill component 290 (via, for example, the coordinator component 240). If the NLU result data includes a single NLU hypothesis, the NLU component 260 may send the NLU result data to the skill component 290 associated with the NLU hypothesis. If the NLU result data includes an N-best list of NLU hypotheses, the NLU component 260 may send the highest scoring NLU hypothesis to the skill component 290 associated with the highest scoring NLU hypothesis. As described above, the NLU component 260 and / or the skill component 290 may use the interaction score to determine text data representing an indication of a handoff from one speech processing system 292 to another.

[0054] Skill components 290 may be software that runs on system 120, similar to software applications. That is, skill components 290 may enable system 120 to perform specific functions, to provide data or to produce some other requested output. System 120 may be configured with more than one skill component 290. For example, a weather service skill component may enable system 120 to provide weather information, a car service skill component may enable system 120 to book a ride with respect to a taxi or ride-sharing service, a restaurant skill component may enable system 120 to order a pizza with respect to a restaurant's online ordering system, and the like. Skill components 290 may operate in coordination between system 120 and other devices (e.g., user device 110) to accomplish certain functions. Input to skill components 290 may come from speech processing interactions or through other interaction or input sources. Skill components 290 may include hardware, software, firmware, etc. that may be dedicated to a particular skill component 290 or shared between different skill components 290.

[0055] The skill system 225 may communicate with a skill component 290 within the system 120 directly and / or via the coordinator component 240. The skill system 225 may be configured to perform one or more actions. A skill may enable the skill system 225 to perform a specific function in order to provide data or perform some other action requested by a user. For example, a weather service skill may enable the skill service 225 to provide weather information to the system 120, a car service skill may enable the skill system 225 to book a ride with respect to a taxi or ride-sharing service, an order pizza skill may enable the skill system 225 to order a pizza with respect to a restaurant's online ordering system, and the like. Additional types of skills include home automation skills (e.g., enabling a user to control home devices such as lights, door locks, cameras, thermostats, etc.), entertainment device skills (e.g., enabling a user to control an entertainment device such as a smart TV), video skills, flash briefing skills, and custom skills that are not associated with any preconfigured skill type. The system 120 may include a skill component 290 that is dedicated to interacting with the skill system 225. A skill, skill device, or skill component may include a skill component 290 operated by the system 120 and / or a skill operated by the skill system 225 .

[0056] The speech processing system manager and / or each speech processing system 292 may include Figure 7 2, which generates audio data (e.g., synthesized speech) from text data using one or more different methods. The text data input to the TTS component 280 can come from the skill component 290, the coordinator component 240, and / or another component of the system. The text data may include instructions for the speech processing component and / or data in response to a command.

[0057] System 120 may include Figure 8 295, which identifies one or more users associated with the data input to the system. The user identification component 295 may take the audio data 211 and / or the text data output by the ASR component 250 as input. The user identification component 295 may perform user identification by comparing the audio characteristics in the input audio data 211 with the stored user audio characteristics. The user identification component 295 may also perform user identification by comparing the biometric data (e.g., fingerprint data, iris data, etc.) received by the system associated with the current user input with the stored user biometric data. The user identification component 295 may also perform user identification by comparing the image data (e.g., including the representation of at least one feature of the user) received by the system associated with the current user input with the stored image data (which includes the representation of the features of different users). The user identification component 295 may perform additional user identification processes, including those known in the art.

[0058] The user identification component 295 determines a score indicating whether the user input originated from a particular user. For example, a first score may indicate the likelihood that the user input originated from a first user, a second score may indicate the likelihood that the user input originated from a second user, etc. The user identification component 295 also determines an overall confidence level regarding the accuracy of the user identification operation.

[0059] The output of the user identification component 295 may include a single user identifier corresponding to the most likely user who initiated the user input. Alternatively, the output of the user identification component 295 may include a NN-best list of user identifiers, where the corresponding scores indicate the likelihood that the corresponding user issued the user input. The output of the user identification component 295 may be used to inform the NLU process and the processing performed by other components of the system. As described above, the interaction determination component 296 can use this user identifier to identify the user account in the profile storage device 270.

[0060] The system 120 may include a profile storage 270. The profile storage 270 may include various information about individual users, groups of users, devices, etc., that interact with the system. A "profile" refers to a set of data associated with a user, device, etc. The data of a profile may include preferences specific to the user, device, etc.; input and output capabilities of the device; Internet connection information; user bibliographic information; subscription information, and other information.

[0061] The profile storage device 270 may include one or more user profiles, each of which is associated with a different user identifier. Each user profile may include various user identification information. Each user profile may also include the user's preferences and / or one or more device identifiers representing one or more devices of the user. When the user logs into an application installed on the user device 110, the user profile (associated with the presented login information) may be updated to include information about the user device 110. As described, the profile storage device 270 may also include data showing the user's interaction history, including commands and the time at which the commands were received. The profile storage device 270 may also include data showing when the second user was present to hear the handover indication of the command issued by the first user.

[0062] The profile storage device 270 may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a family profile associated with user profiles associated with multiple users of a single family. A group profile may include preferences shared by all user profiles associated therewith. Each user profile associated with a group profile may additionally include preferences specific to the user associated therewith. That is, each user profile may include preferences that are unique to one or more other user profiles associated with the same group profile. A user profile may be an independent profile or may be associated with a group profile.

[0063] Profile storage device 270 may include one or more device profiles. Each device profile may be associated with a different device identifier. Each device profile may include various device identification information. Each device profile may also include one or more user identifiers representing one or more users associated with the device. For example, a profile for a family device may include user identifiers for family users.

[0064] The system may be configured to incorporate user permissions, and the activities disclosed herein may be performed only with the approval of the user. As described above, these permissions may include permission (or denial) to use a particular speech processing system 292. Thus, the systems, devices, components, and techniques described herein may be configured to limit processing where appropriate, and process user information only in a manner that ensures compliance with all appropriate laws, regulations, standards, etc. The systems and techniques may be implemented on a geographic basis to ensure compliance with laws in various jurisdictions and entities where components of the system and / or users are located.

[0065] Figure 3A and Figure 3B A system for processing speech according to an embodiment of the present invention is shown. Figure 3A , (except for the above reference Figure 2A or Figure 2B The first speech processing system 292a also includes a first ASR component 250a, a first NLU component 260a, a first TTS component 280a, and / or a first skill component 290a. (In addition to some or all other components described above, the first speech processing system 292a also includes a first ASR component 250a, a first NLU component 260a, a first TTS component 280a, and / or a first skill component 290a. Figure 2A or Figure 2B The second speech processing system 292b also includes a second ASR component 250b, a second NLU component 260b, a second TTS component 280b, and / or a second skill component 290a in addition to some or all other components described herein. Any number of additional speech processing systems 292 are within the scope of the present disclosure. In some embodiments, the first speech processing system 292a and the second speech processing system 292b share the same ASR component 250, NLU component 260, TTS component 280, and / or skill component 290.

[0066] As described above, the first speech processing component 292a receives input data 311 from the user device 110. The input data 311 can be or include audio data 211, and can be a single phrase (e.g., "Alexa, what time is it?") or a representation of one or more phrases of a multi-turn conversation interaction with a user (e.g., "Alexa, make a restaurant reservation," "Tuesday at 8 pm," and "Italian"). The first speech processing component 292a determines output data 312, which can be or include some or all of the output audio data 212, as described herein, and sends the output data to the user device 110 for output. The output data 312 can similarly represent a single phrase of synthesized speech (e.g., "three o'clock") or can be one or more phrases of a multi-turn conversation (e.g., "OK, what time?" and "OK, what type of restaurant?").

[0067] As described above, the speech processing system manager can process the input data 311 to determine the first speech processing system 292a (a default system, such as a system associated with a user input (e.g., a button press) of the user device 110 and / or a system associated with a wake word) and determine the second speech processing system 292b (e.g., indicated by at least one output of the NLU component 260). Upon receiving the input data 311, the first speech processing system 292a can determine the first output data 312 that first confirms the receipt, such as a beep, pitch, or other such sound.

[0068] The first speech processing system 292a can process the input data 311 (using, for example, the NLU component 260) to determine the command represented in the input data 311 and can determine command data 302 corresponding to the command. The command data 302 can be, for example, text data representing the words in the input data 311 (e.g., text data output by the ASR component 250) and / or a domain, intent, and / or entity corresponding to the command (as, for example, output by the NLU component 260). The command data 302 can alternatively or additionally include an API call corresponding to the second speech processing system 292b. In other embodiments, the command data 302 includes a representation of the input audio data 211.

[0069] For example, if the input data 311 includes a representation of the phrase "Alexa, roll down the window," the first speech processing system 292a may process the input data 311 to determine corresponding text data and determine that the domain corresponding to the input data 311 is, for example, "car." The speech processing system 292a may further determine that the intent corresponding to the input data 311 is "change car state" and that the entities associated with the input data 311 are "window" and "down." Depending on the particular second speech processing system 292b and its requirements for communication, the first speech processing system 292a may include in the command data 302 the input data 311 itself, the audio data 211, text data corresponding to the input data 311, domains, intents, and / or entities, and / or an API call to a window control API. Any type of command data 302 corresponding to a command represented in the input data 311 is within the scope of the present disclosure.

[0070] The second speech processing system 292b can then process the command data 302 to determine the response data 304. The processing of the command data 302 by the second speech processing system 292b can be similar to the way the first speech processing system 292a processes the input data 311, as described above. For example, if the command data 302 includes some or all of the input data 311, the ASR component 250 of the second speech processing system 292b can determine the corresponding text data, the NLU component 260 of the second speech processing system 292a can determine the corresponding domain, and so on. If the command data 302 includes text data, the NLU component 260 can process the command data 302 without the ASR component 250. If the command data 302 includes NLU output data and / or API calls, the coordinator component 240 of the second speech processing system 292b can process the command data 302, select a corresponding skill component 290, and send the command data 302 to the selected skill component 290. The skill component 290 can then process the command data (using, for example, a dialog model or a natural language generation component) to determine response data 304 .

[0071] In some embodiments, the first speech processing system 292a sends the command data 302 to the second speech processing system 292b, receives the response data 304 in response, and then processes the response data 304 to determine the output data 312, which can be or include the output audio data 212. As described above, the response data 304 can include an indication of the second speech processing system 292b. This indication can be the name of the second speech processing system 292b, the voice style of the second speech processing system 292b, and / or one or more words or wording styles associated with the second speech processing system 292b. However, as also described above, the user may not know or care that the second speech processing system 292b is involved in processing the input data 311, and therefore may prefer to receive a response (in the form of output data 312) that shows the indication of the first speech processing system 292a (e.g., the name of the first speech processing system 292a and / or the speaking / word style of the first speech processing system 292a). Thus, the first speech processing system 292a may process the response data 304 to remove any indication of the second speech processing system 292b and include the indication of the first speech processing system 292a.

[0072] For example, if the response data 304 includes audio data, this audio data may include representations of one or more words spoken in the characteristics of the second speech processing system 292b. Thus, the first speech processing system 292a may process the audio data (e.g., using the ASR component 250) to determine corresponding text data, and then process the text data (using, for example, the TTS component 280) to determine audio data, which includes representations of one or more words spoken in the characteristics of the first speech processing system 292a. Then, even though the second speech processing system 292b processes the command data 302, the user 5 may hear the output audio 12 corresponding to the characteristics of the first speech processing system 292a.

[0073] If the response data 304 includes text data corresponding to the response, the text data may include an indication of the second speech processing system 292b, such as the name of the second speech processing system 292b, a word or phrase associated with the second speech processing system 292b, and / or a sentence structure or grammatical style associated with the second speech processing system 292b. Thus, the first speech processing system 292a may process the text data of the response data 304 to remove the indication of the second speech processing system 292b by, for example, paraphrasing the text data. Thus, the first speech processing system 292a may determine output data 312 that includes a representation of the response data 304, but corresponds to the characteristics of the first speech processing system 292a.

[0074] For each item of input data 311 received and processed, the first speech processing system 292a may send one or more items of command data 302 to the second speech processing system 292b and receive one or more items of response data 304. That is, if the user issues a command as represented in the input data 311, the first speech processing system 292a may send a single corresponding item of command data 302 (and receive a single item of response data 304 in response) or may send multiple items of command data 302 (and receive multiple items of response data 304 in response). Thus, the first speech processing system 292a may cause multiple rounds of sending command data 302 and receiving response data 304 for a single command represented in the input data 311. Similarly, the response data 304 may correspond to additional information required to prompt the user to execute the command, and the first speech processing system 292a and / or the second speech processing system 292b may therefore cause multiple rounds of dialogue, as represented by the output data 312.

[0075] In some embodiments, such as Figure 3BAs shown, the first voice processing system 292a can stop processing the first input data A 311a and cause the second voice processing system 292b to process the second input data B 311b. For example, this process, described herein as a handover process, can occur when the first voice processing system 292a determines that the second input audio B 311b may include a representation of private, privileged, or other sensitive information corresponding to the second voice processing system 292b rather than the first voice processing system 292a. For example, if the first input data A 311a includes a representation of the command "What is my checking account balance?", the first voice processing system 292a can determine that the bank-related second voice processing system 292b should then process the second input data B 311b because it may contain personal information, such as a PIN or password associated with the second voice processing system 292b. Therefore, the first voice processing system 292a can send the command data 310 that may include a handover signal to the second voice processing system 292b when making this determination and / or identifying the second voice processing system 292b. The handover signal may be command data that causes the second speech processing system 292b to start processing input data A 311a and / or additional input data B 311b and determine and cause output of output data B 312b based thereon. After sending the command data 310, the first speech processing system 292a may stop processing the input data A 311a and / or additional input data B 311b. Then, the second speech processing system 292b may start processing the input data B 311b upon receiving the command data 310. The first speech processing system 292a may also send additional data corresponding to the input data A 311a to the second speech processing system 292b, such as corresponding text data, domains, intents and / or entities associated with the input data A 311a, and / or the input data A 311a itself. The first output data A 312b determined by the first speech processing system 292a may include an indication of the handover, such as a name associated with the second speech processing system 292b.

[0076] Figure 4A , Figure 4B and Figure 4C is a flow chart showing the use of a speech processing system according to an embodiment of the present disclosure. Figure 4A, the user device 110 sends input data 311 to the first natural language system 492a, which may be or include the first speech processing system 292a. The input data 311 may be or include input audio data 211, and / or may be or include other input data, such as video data, text data, and / or other such data. As described herein, the user device 110 may identify the first speech processing system 292a based on the associated wake-up word represented in the input data 311. In other embodiments, the first speech processing system 292a may be the default speech processing system to which the user device 110 sends all input data 311; in some embodiments, the default first speech processing system 292a may then determine and activate another first speech processing system 292a to process the input data 311.

[0077] As also described herein, the first speech processing system 292a determines (402) a command represented in the input data 311 using, for example, ASR and / or NLU techniques. Based at least in part on the text data determined by the ASR component 250 and / or the NLU data determined by the NLU component 260, the first speech processing system 292a determines (404) that the command corresponds to a second natural language system 492b, which may be or include the second speech processing system 292b. The first speech processing system 292a may then determine command data, which, as described above, may include some or all of the input audio data 311, text data corresponding to the input data 311, NLU data corresponding to the text data, and / or one or more API calls to an API associated with the second speech processing system 292b. The first speech processing system 292a may then send the command data 302 to the second speech processing system 292b.

[0078] The second speech processing system 292b may then process the command data to determine (408) response data 304. As described above, the response data 304 may include an indication of the second speech processing system 292b, such as the name of the second speech processing system 292b, the speech style of the second speech processing system 292b, and / or the grammatical style of the second speech processing system 292b. The speech style and / or the grammatical style may correspond to the characteristics of the second speech processing system 292b. The second speech processing system 292b may then send the response data 304 to the first speech processing system 292a.

[0079] Then, the first speech processing system 292a can process the response data 304 to determine (412) output data 312 representing the second response. As described above, the output data 312 may include an indication of the first speech processing system 292a, such as the name of the first speech processing system 292a, the voice style of the first speech processing system 292a, and / or the grammatical style of the second speech processing system 292a. Therefore, in some embodiments, although the second speech processing system 292b is used to process the input data 311, the first speech processing system 292a does not include an indication of the second speech processing system 292c in the output data 312, and therefore, the user may not know that the second speech processing system 292b is so used. The first speech processing system 292a can send the output data 312 to the user device 110 and thus cause output thereon.

[0080] refer to Figure 4B , in some embodiments, the first speech processing system 292a can determine an output that causes an indication of the second speech processing system 292b. As described herein, the first speech processing system 292a can receive (450) input data 311 and determine (452) whether the command represented in the input data 311 corresponds to the second speech processing system 292b. If not, the first speech processing system 292a can process (454) the input data 311 and determine a response (as described herein). For example, a user speaks a wake word for the first speech processing system 292a (e.g., "Alexa") and a command associated with the first speech processing system 292a (e.g., "What time is it?"). The first speech processing system 292a then processes the input data 311 to determine a response to the command (e.g., "2 p.m.").

[0081] In other embodiments, the first voice processing system 292a can determine (456) that the input data 311 and / or the response to the input data 311 may include a representation of private, privileged and / or sensitive information. In these embodiments, the first voice processing system 292a can hand over the processing (466) of the input data 311 to the second voice processing system 292b, and then stop the processing of the input data 311. This information can be, for example, bank account information, medical information, user name, password or other such information. The user can allow the second voice processing system 292b to access this information, but may not want the first voice processing system 292a to access this information. The voice processing system 292a can cause the output of the audio indicating this handover to the second voice processing system 292b, so that the user knows that the second voice processing system 292b (rather than the first voice processing system 292a) is being used to process the input data 311.

[0082] When the first speech processing system 292a determines that the output data 312 may include facts, opinions, and / or sensitive information that may be associated with the second speech processing system 292b, the first speech processing system can similarly hand over processing to the second speech processing system 292b. Thus, the first speech processing system 292a can hand over processing so that it is clear to the user that the output data 312 was determined by the second speech processing system 292b rather than the first speech processing system 292a. For example, if the input data 311 includes a representation of the command "What is the best restaurant in my area?", the first speech processing system 292a can determine that the input data 311 should be handed over to the second restaurant-specific speech processing system 292b for further processing so that the user knows that the second speech processing system 292a, rather than the first speech processing system 292, selected the "best" restaurant.

[0083] As described herein, in some embodiments, as part of processing (462) input data 311, the first speech processing system 292a can communicate with the second speech processing system 292b, and the indication of the second speech processing system 292b may not be output. However, in some embodiments, the first speech processing system 292a can determine to output the indication. For example, the first speech processing system 292a can determine (458) that the command corresponds to a response that occurs in the future (e.g., at least five minutes in the future). For example, such a command can be to set an alarm, request a notification of an event occurrence, schedule a meeting or reminder, set an early warning such as a weather warning, or other such future actions. If the second speech processing system 292 is used to process the input data 311 and complete the future action, the second speech processing system 292b can cause the audio corresponding to the action to be output to the user. Then, the user may be confused because the command to create the future action is for the first speech processing system 292a, but the execution of the action occurs using the second speech processing system 292b. The first speech processing system 292a can therefore cause the indication of the second speech processing system 292b to be output.

[0084] For example, input data 311 may include a representation of the command "SmartCar, set an alarm for 7pm". The SmartCar voice processing system may determine that this command is associated with the Alexa voice processing system and communicate with the Alexa voice processing system using the command data and response data described above. Based on this communication, the Alexa voice processing system may set an alarm and output audio corresponding to the alarm when appropriate. For example, the SmartCar voice processing system may therefore output audio corresponding to the phrase "Alexa will set an alarm for 7pm".

[0085] If the first speech processing system 292a determines (460) that some or all of the output data 312 (and / or other output data) will be output using a device different from the user device 110, the first speech processing system may alternatively or additionally cause the output of an indication of the second speech processing system 292b. Using such a different device may also cause confusion to the user. The first speech processing system 292a may alternatively or additionally cause the output (464) (or otherwise not cause the output) of an indication of the second speech processing system 292b based on user preferences in a user account associated with the user device 110. The user account may also include an indication that a permission request should be output before processing with the second speech processing system 292b is allowed. The first speech processing system 292a and / or the second speech processing system 292b may similarly determine to output a permission request based on a command.

[0086] In some embodiments, the first voice processing system 292a may determine that the response to the received command includes a response determined by the first voice processing system 292a and a response from the second voice processing system 292. In other embodiments, the first voice processing system 292a may determine that the response includes output from the user device 110 and / or other user devices. This response may or may not include a response from the first voice processing system 292a. For example, if the command is "Alexa, turn off the lights", the first voice processing system 292a may instruct the lights near the user to turn off, but no audio output will be caused. In these embodiments, the command data 310 may include instructions, such as API calls, but may not include additional information corresponding to the input data 311. In other words, the first voice processing system 292a may simply instruct the user device to act according to the command. Other user devices may output non-voice audio to indicate that a command has been received; this non-voice audio may be, for example, a beep or similar sound.

[0087] like Figure 4CAs shown, the first speech processing system 292a may incorporate some or all of the response data 304 into the output data 312. As described above, the first speech processing system 292a may receive (480) the response data 304 from the second speech processing system 292b. The first speech processing system 292a may then paraphrase (482) some or all of the response data 304. For example, the first speech processing system 292a may determine that the response data 304 includes an indication of the second speech processing system 292b, such as the name of the second speech processing system 292b, words associated with the second speech processing system 292b, and / or a style of wording associated with the second speech processing system 292b. Therefore, the first speech processing system 292a may process the response data 304 using, for example, the NLU component 260 to determine (486) the paraphrased response data. For example, the NLU component can determine the domain, intent, and / or entity associated with the response data 480, and then cause the skill component 290, such as the natural language generation component, to create rephrased response data based on the domain, intent, and / or entity. In other embodiments, the first speech processing system 292a can use (484) the response data as output data 212. In some embodiments, the first speech processing system 292a can combine (488) some or all of the response data 304 with some or all of the rephrased response data to create (492) output data 312. In other embodiments, the first speech processing system 292a can use (490) only the rephrased response data as output data 312.

[0088] Figure 5 260 (e.g., Figure 2A or Figure 2B 260 attempts to semantically interpret the text represented in the text data. That is, the NLU component 260 determines the meaning behind the text represented in the text data on a word-by-word basis. The NLU component 260 interprets the text to derive the user's intent or desired action and relevant information segments in the text that allow a device (e.g., user device 110, system 120, skill 290, and / or skill system 225) to complete the action.

[0089] The NLU component 260 can process text data that includes several hypotheses for a single utterance. For example, if the ASR component 250 outputs an ASR result that includes an N-best list of hypotheses, the NLU component 260 can process the text data with respect to all (or part of) the text interpretations represented therein. The NLU component 260 can annotate the text represented in the text data by parsing and / or marking the text. For example, for the text "Tell me the weather in Seattle", the NLU component 260 can mark "Tell me the weather in Seattle" as a command (e.g., to output weather information) and mark "Seattle" as the location of the weather information.

[0090] The NLU component 260 can include a shortlister component 550. The shortlister component 550 selects applications that can be executed relative to the text data 610 input to the NLU component (e.g., applications that can execute commands). Thus, the shortlister component 550 limits downstream, more resource-intensive NLU processing to applications that can execute commands.

[0091] Without the candidate lister component 550, the NLU component 260 can process a given hypothesis with respect to each application of the system in parallel, serially, or using some combination thereof. By including the candidate lister component 550, the NLU component 260 can process a given hypothesis only with respect to the application that can execute the command. This reduces the overall computing power and latency attributed to the NLU processing.

[0092] The NLU component 260 may include one or more recognizers 563a-n. Each recognizer 563 may be associated with a different function, content source, and / or speech processing system. The NLU component 260 may determine the function that may be associated with a command represented in the text data input thereto in order to determine the appropriate recognizer 563 to process the hypothesis. The NLU component 260 may determine that a command represented in the text data may be associated with more than one function. Multiple recognizers 563 may be functionally associated (e.g., a phone / communication recognizer and a calendar recognizer may utilize data from the same contact list).

[0093] If the candidate lister component 550 determines that the text corresponding to the hypothesis may be associated with multiple skills 290 and / or multiple speech processing systems, the recognizers 563 associated with the skills 290 and / or multiple speech processing systems (e.g., recognizers 563 associated with applications in the subset selected by the candidate lister 550) can process the text. The selected recognizers 563 can process the text in parallel, serially, partially in parallel, etc. For example, if the text corresponding to the hypothesis may involve both a communication speech processing system and a music speech processing system, the recognizer associated with the communication speech processing system can process the text in parallel or partially in parallel with the recognizer associated with the music application that processes the text. The output generated by each recognizer 563 can be scored, with the output from all recognizers 563 having the highest overall score typically being selected as the correct result.

[0094] The NLU component 260 may communicate with various storage devices to determine possible speech processing systems associated with commands represented by text data. The NLU component 260 may communicate with an NLU storage device 573, which includes a device database (574a to 574n) that identifies functions associated with a particular device. For example, the user device 110 may be associated with a speech processing system for music, calendar, contact list, device-specific communications, etc. In addition, the NLU component 260 may communicate with an entity library 582, which includes database entries about specific services on a specific device, indexed by device ID, user ID, or group user ID, or some other indicator.

[0095] Each recognizer 563 may include a named entity recognition (NER) component 562. The NER component 562 attempts to identify grammatical and lexical information that can be used to interpret the meaning of commands represented in the text data input therein. The NER component 562 identifies portions of text represented in the text data input into the NLU component 260 that correspond to named entities recognizable by the system. The NER component 562 (or other components of the NLU component 260) can also determine whether a word refers to an entity that is not explicitly mentioned in the utterance text, such as a word such as "he," "she," or "it."

[0096] Each recognizer 563, and more specifically, each NER component 562 can be associated with a specific grammar model 576, a specific intent set 578, and a specific featured dictionary 586. Each gazetteer 584 can include functional index dictionary information associated with a specific user and / or device. For example, gazetteer A (584a) includes functional index dictionaries 586aa to 586an. For example, a user's music functional dictionary information may include album title, artist name, and song name, while the user's contact list dictionary information may include contact name. Since each user's music collection and contact list may be different, this featured information improves entity resolution.

[0097] The NER component 562 can apply a grammar model 576 and / or a dictionary 586 (associated with a recognizer 563 implementing the NER component 562) associated with a function to determine one or more entities mentioned in the text data input thereto. In this way, the NER component 562 can identify "slots" (i.e., specific words in the text data) that may be needed for subsequent command processing. The NER component 562 can also mark each slot with different levels of specificity (e.g., names of people, places, cities, names of artists, song titles, etc.).

[0098] Each grammar model 576 may include the names of entities (i.e., nouns) that are commonly found in text related to a particular function associated with the grammar model 576, while the lexicon 586 is specific to the user and / or the user device 110 from which the input data 311 or input text data originated. For example, a grammar model 576 associated with a shopping function may include a database of words that people commonly use when discussing shopping.

[0099] A process called named entity resolution can associate portions of text to entities known to the system. To perform this named entity resolution, the NLU component 260 can use gazetteer information (584a to 584n) stored in the entity library storage device 582. Gazetteer information 584 can be used to match text represented in the text data to different entities (e.g., song titles, contact names, etc.). Gazetteers can be associated with users (e.g., a particular gazetteer can be associated with a particular user's music collection), can be associated with certain functional categories (e.g., shopping, music, video, communication, etc.), or can be organized in various other ways.

[0100] Each recognizer 563 may also include an intent classification (IC) component 564. The IC component 564 parses the text data to determine the intent of the function associated with the recognizer 563 that may correspond to the text data. The intent corresponds to an action to be performed in response to a command represented by the text data. The IC component 564 can communicate with a database 578 of words associated to the intent. For example, a music intent database can associate words and phrases such as "quiet", "volume off" and "mute" to the "mute" intent. The IC component 564 identifies potential intents by comparing the words in the text data with the words and phrases in the intent database 578, which is associated with the function associated with the recognizer 563 that implements the IC component 564.

[0101] Intents recognizable by a particular IC component 564 can be associated to a function-specific (i.e., a function associated with the recognizer 563 that implements the IC component 564) grammar model 576 having "slots" to be filled. Each slot of the grammar model 576 can correspond to a portion of text data that the system considers to correspond to an entity. For example, a grammar model 576 corresponding to the intent of <play music> might correspond to a text data sentence structure such as "play {artist name}", "play {album name}", "play {song name}", "play {song name} by {artist name}", etc. However, to make parsing more flexible, the grammar model 576 may not be structured as sentences, but rather based on associating slots with grammar tags.

[0102] For example, NER component 562 can parse text data to identify words as subjects, objects, predicates, prepositions, etc. based on grammatical rules and / or models before identifying named entities in text data. IC component 564 (implemented by the same recognizer 563 as NER component 562) can use the recognized predicate to identify intent. NER component 562 can then determine the grammatical model 576 associated with the recognized intent. For example, the grammatical model 576 for the intent corresponding to <play music> can specify a slot list and any object modifiers (e.g., prepositional phrases) suitable for playing the recognized object, such as {artist name}, {album name}, {song name}, etc. Then, NER component 562 can search for corresponding fields in the dictionary 586 associated with the function associated with the recognizer 563 that implements NER component 562, and can match words and phrases in the text data previously marked as grammatical objects or object modifiers by NER component 562 with those words and phrases recognized in the dictionary 586.

[0103] NER component 562 can perform semantic tagging, which is to mark a word or a combination of words according to the type / semantic meaning of the word. NER component 562 can use heuristic grammar rules to parse text data, or can use technologies such as hidden Markov model, maximum entropy model, log linear model, conditional random field (CRF) to build a model. For example, the NER component 562 implemented by the music function identifier 563 can parse and mark the text corresponding to "playing the Rolling Stones' mom's little helper" as {predicate}: "play", {object}: "mom's little helper", {object preposition}: "by", and {object modifier}: "Rolling Stones". NER component 562 can identify "play" as a predicate based on a word database associated with a music function, and IC component 564 (which can also be implemented by the music function identifier 563) can determine that the word corresponds to <playing music> intention. At this stage, the meaning of "mother's little helper" and "the Rolling Stones" may not have been determined, but based on grammatical rules and models, the NER component 562 may have determined that the text of these phrases is related to the grammatical objects (i.e., entities) of the text data.

[0104] Then the framework associated to the intent can be used to determine which database fields can be searched to determine the meaning of these phrases, such as searching the user's gazetteer 584 to obtain similarities with the framework slot. For example, the framework for the <play music> intent may indicate an attempt to parse the identified object based on {artist name}, {album name} and {song name}, and another framework for the same intent may indicate an attempt to parse the object modifier based on {artist name}, and parse the object according to {album name} and {song name} associated to the identified {artist name}. If the search of the gazetteer 584 does not use the gazetteer information to parse the slot / field, the NER component 562 can search the database of the common words associated with the function in the knowledge base 572. For example, if the text data includes text corresponding to "play the song of the Rolling Stones", then after failing to determine the album name or song name of the "song" called "the Rolling Stones", the NER component 562 can search the vocabulary of the function to find the word "song". In some embodiments, the common words can be checked before the gazetteer information, or both can be tried, which may produce two different results.

[0105] The NLU component 260 can tag text to give meaning to the text. For example, the NLU component 260 can tag "Play Mom's Little Helper by the Rolling Stones" as {Intent}: <Play Music>, {Artist Name}: The Rolling Stones, {Media Type}: Song, and {Song Title}: Mom's Little Helper. In another example, the NLU component 260 can tag "Play Song by the Rolling Stones" as {Intent}: <Play Music>, {Artist Name}: The Rolling Stones, and {Media Type}: Song.

[0106] The candidate lister component 550 may receive the candidate list from the ASR component 250 (eg, Figure 6 The ASR component 250 may use sentence embedding techniques to embed the text data 610 into a form that can be processed by the training model. Sentence embeddings may include text in the text data 610 in a structured form that enables the training model of the candidate lister component 550 to operate on the text. For example, the embedding of the text data 610 may be a vector representation of the text data.

[0107] The candidate lister component 550 can make a binary determination (e.g., a yes or no determination) as to which skills 290 are associated with the text data 610. The candidate lister component 550 can make such a determination using one or more of the training models described above. If the candidate lister component 550 implements a single training model for each skill 290, the candidate lister component 550 can simply run the model associated with the enabled application, as indicated in the profile associated with the user device 110 and / or user that initiated the command.

[0108] The candidate lister component 550 can generate N-best list data representing applications that can be executed with respect to the command represented in the text data 610. The size of the N-best list represented in the N-best list data is configurable. In one example, the N-best list data can indicate each application of the system and contain an indication for each application regarding whether the application is likely to be able to execute the command represented in the text data 610. In another example, instead of indicating each application of the system, the N-best list data can only indicate all applications that are likely to be able to execute the command represented in the text data 610. In yet another example, the candidate lister component 550 can implement threshold processing so that the N-best list data can indicate no more than a maximum number of applications that can execute the command represented in the text data 610. In one example, the threshold number of applications that can be represented in the N-best list data is ten (10). In another example, the applications included in the N-best list data may be limited by a threshold score, wherein only applications indicating a likelihood of processing a command are above a certain score (as determined by processing the text data 610 by the candidate lister component 550 with respect to such applications).

[0109] The NLU component 260 may compile the data output by each recognizer 563 that processes the text data input to the NLU component 260 into a single N-best list, and may send N-best list data 640 (representing the N-best list) to a pruning component 650 (eg, Figure 6 563). Each entry in N-best list data 640 may correspond to a markup text output by a different recognizer 563. Each entry in N-best list data 640 may be associated with a corresponding score indicating that the markup text corresponds to a function associated with the recognizer 563 from which the markup text was output. For example, N-best list data 640 may be represented as:

[0110] [0.95] Intent: <Play music> Source: Alexa, SmartCar

[0111] [0.70] Intent: <Roll down windows> Source: SmartCar

[0112] [0.01] Intent: <Navigate> Source: Alexa, SmartCar

[0113] [0.01] Intent: <Play video> Source: Alexa

[0114] The pruning component 650 creates a new, shorter N-best list (ie, represented in the N-best list data 660 discussed below) based on the N-best list data 640. The pruning component 650 can sort the marked text represented in the N-best list data 640 according to their respective scores.

[0115] The pruning component 650 can perform score thresholding with respect to the N-best list data 640. For example, the pruning component 650 can select entries associated with scores that meet (e.g., meet and / or exceed) a score threshold represented in the N-best list data 640. The pruning component 650 can also or alternatively perform entry quantity thresholding. For example, the pruning component 650 can select the highest scoring entries associated with each different functional category (e.g., music, shopping, communication, etc.) represented in the N-best list data 640, wherein the new N-best list data 660 includes the total number of entries that meet or are below the entry threshold number. The purpose of the pruning component 650 is to create a new list of the highest scoring entries so that downstream, more resource-intensive processes can operate only on the marked text entries that are most likely to correspond to commands input to the system.

[0116] The NLU component 260 may also include a lightweight slot filling component 652. The lightweight slot filling component 652 may obtain text from the slots represented in one or more tagged text entries output by the deletion component 650 and modify it to make the text easier to process by downstream components. The lightweight slot filling component 652 may perform low-latency operations that do not involve heavy operations such as reference to a knowledge base. The purpose of the lightweight slot filling component 652 is to replace words with other words or values ​​that may be easier for downstream components to understand. For example, if the tagged text entry includes the word "tomorrow", the lightweight slot filling component 652 may replace the word "tomorrow" with an actual date for downstream processing. Similarly, the lightweight slot filling component 652 may replace the word "CD" with the word "album" or the word "disc". The replaced words are then included in the N-best list data 660.

[0117] The NLU component 260 sends the N-best list data 660 to the entity resolution component 670. The entity resolution component 670 can apply rules or other instructions to standardize the tags or tokens from the previous stage into intent / slot representations. The exact transformation can depend on the function (for example, for the travel function, the entity resolution component 670 can transform the text mention of "Boston Airport" into the standard BOS three-letter code referring to the airport). The entity resolution component 670 can refer to an authoritative source (e.g., a knowledge base) that is used to specifically identify the precise entity referenced in each slot of each marked text entry represented in the N-best list data 660. A specific intent / slot combination can also be associated with a specific source, which can then be used to parse the text. In the example "Play the Rolling Stones song", the entity resolution component 670 can refer to a personal music catalog, an Amazon music account, a user profile (described herein), etc. The entity resolution component 670 can output data including a modified N-best list that is based on the N-best list represented in the N-best list data 660, but also includes more detailed information about the specific entity mentioned in the slot (e.g., entity ID) and / or more detailed slot data that can ultimately be used by the function. The NLU component 260 can include multiple entity resolution components 670 and each entity resolution component 670 can be specific to one or more functions.

[0118] Entity resolution component 670 may not be able to successfully resolve each entity and fill each slot represented in N-best list data 660. This may cause entity resolution component 670 to output incomplete results. NLU component 260 may include final sorter component 690, which may consider such errors when determining how to sort the marked text entries for possible execution. For example, if book function identifier 563 outputs a marked text entry including a <read book> intention sign, but entity resolution component 670 cannot find a book that matches the title with the project text, then final sorter component 690 can re-score a specific marked text entry that has a lower score. Final sorter component 690 can also assign a specific confidence to each marked text entry input therein. The confidence score of a specific marked text entry may be affected by whether the marked text entry has an unfilled slot. For example, if the marked text entry associated with the first function includes all filled / parsed slots, the marked text entry may be associated with a higher confidence than another marked text entry including at least some unfilled / unparsed slots.

[0119] The final sorter component 690 can apply re-scoring, biasing or other techniques to obtain the most preferred tagged and parsed text entries. To this end, the final sorter component 690 can consider not only the data output by the entity resolution component 670, but also other data 691. Other data 691 can include various information. For example, other data 691 can include function ratings or popularity data. For example, if a function has a particularly high rating, the final sorter component 690 can increase the score of one or more tagged text entries associated with or otherwise calling the specific function. Other data 691 can also include information about functions that have been specifically enabled by the user. For example, compared with the tagged text entries associated with or otherwise calling the non-enabled functions, the final sorter component 690 can assign a higher score to the tagged text entries associated with or otherwise calling the enabled functions. User history can also be considered, such as whether the user regularly uses a specific function or uses a specific function at a specific time of the day. Date, time, location, weather, type of user device 110, user ID, context and other information can also be considered. For example, the final sequencer component 690 can take into account when any particular function is currently active (e.g., music is playing, a game is being played, etc.). After the final sequence, the NLU component 260 can output NLU output data 685 to the coordinator component 240. The NLU output data 685 can include various entries, where each entry represents an NLU processing confidence score, an intent, slot data, and potential skills or skills that may operate with respect to the data of the corresponding entry.

[0120] After preliminary ranking, NLU component 260 can output NLU result data 685. NLU component 260 can send NLU result data 685 to coordinator component 240. NLU result data 685 can include: first NLU result data 685a, which includes tagged text associated with the first speech processing system; second NLU result data 685b, which includes tagged text associated with the second speech processing system, etc. NLU result data 685 can include tagged text data corresponding to the highest-scoring tagged text entry determined by preliminary ranker component 690.

[0121] The data 685 output from the NLU component 260 may include an N-best list of NLU results, where each item in the N-best list may correspond to a particular recognizer 563 and a corresponding skill 290. Thus, for example, a first NLU result of the N-best list may be associated with a first skill 290a, a second NLU result of the N-best list may be associated with a second skill 290b, a third NLU result of the N-best list may be associated with a third skill 290c, and so on. In addition, the first NLU result may correspond to text tagged with an attribute meaning that enables the first skill 290a to be performed relative to the first NLU result, the second NLU result may correspond to text tagged with an attribute meaning that enables the second skill 290b to be performed relative to the second NLU result, the third NLU result may correspond to text tagged with an attribute meaning that enables the third skill 290c to be performed relative to the third NLU result, and so on. The data 685 may also include a score corresponding to each item in the N-best list. Alternatively, the NLU result data 685 output to a specific skill 290 may include the NER and IC data output by the identifier 563 for the specific skill, while the NLU result data 685 output to the coordinator component 240 may include only a portion of the NLU result data 685, such as scores corresponding to certain skills.

[0122] The system can be configured with thousands, tens of thousands, etc. of skills 290. The coordinator component 240 enables the system to better determine the best skill 290 to execute the command input to the system. For example, a first NLU result may correspond to or substantially correspond to a second NLU result, even if the first NLU result is operated by a first skill 290a and the second NLU result is operated by a second skill 290b. The first NLU result may be associated with a first confidence score indicating the confidence of the system regarding the NLU processing performed to generate the first NLU result. In addition, the second NLU result may be associated with a second confidence score indicating the confidence of the system regarding the NLU processing performed to generate the second NLU result. The first confidence score may be similar or identical to the second confidence score because the first NLU result corresponds to or substantially corresponds to the second NLU result. The first confidence score and / or the second confidence score may be a numerical value (e.g., from 0.0 to 1.0). Alternatively, the first confidence score and / or the second confidence score may be a grouping value (e.g., low, medium, high).

[0123] The coordinator component 240 may request the first skill 290a and the second skill 290b to provide potential result data based on the first NLU result and the second NLU result, respectively. For example, the coordinator component 240 may send the first NLU result to the first skill 290a together with a request for the first skill 290a to at least partially execute a command regarding the first NLU result. The coordinator component 240 may also send the second NLU result to the second skill 290b together with a request for the second skill 290b to at least partially execute a command regarding the first NLU result. The coordinator component 240 receives the first result data generated from the execution of the first skill relative to the first NLU result from the first skill 290a. The coordinator component 240 also receives the second result data generated from the execution of the second skill relative to the second NLU result from the second skill 290b.

[0124] Result data 685 may include various components. For example, result data 685 may include content to be output to the user (e.g., audio data, text data, and / or video data). Result data 685 may also include a unique identifier (ID) used by remote system 120 and / or skill server 225 to locate data to be output to the user. Result data 685 may also include instructions. For example, if the command corresponds to "turn on the lights," result data 685 may include instructions to cause the system to turn on the lights associated with user device 110 and / or the user's profile.

[0125] The coordinator component 240 may associate the intent in the NLU result data 685 with the skills 290 before sending the NLU result data 685 to the coordinator component 240. For example, if the NLU result data 685 includes the <play music> intent, the coordinator component 240 may associate the NLU result data 685 with one or more skills 290 that can perform the <play music> intent. Thus, the coordinator component 240 may send the NLU result data 685 paired with the skills 290 to the coordinator component 240. In response to input text data corresponding to “What should I make for dinner today”, the coordinator component 240 may generate a skill pair 290 with associated intents corresponding to:

[0126] Skill 1 / <Roll Down Window>

[0127] Skill 2 / <Start Navigation>

[0128] Skill 3 / <Play Music>

[0129] Systems that do not use a coordinator component 240 as described above may instead select the highest scoring preliminary ranked NLU result data 685 associated with a single skill. The system may send the NLU result data 685 to the skill 290 along with a request for output data. In some cases, the skill 290 may be unable to provide output data to the system. This results in the system indicating to the user that the command cannot be processed, even though another skill associated with a lower-ranked NLU result data 685 may have provided output data in response to the command.

[0130] Figure 7 Components of the system that can be used to perform unit selection, parametric TTS processing, and / or model-based audio synthesis are shown in FIG. Figure 7 As shown, the TTS component / processor 780 may include a TTS front end 716, a speech synthesis engine 718, a TTS unit storage 772, a TTS parameter storage 780, and a TTS back end 734. Among other things, the TTS unit storage 772 may include voice lists 778a-288n, which may include pre-recorded audio segments (referred to as units) used by the unit selection engine 730 when performing unit selection synthesis as described below. Among other things, the TTS parameter storage 780 may include parameter settings 768a-268n, which may be used by the parameter synthesis engine 732 when performing parameter synthesis as described below. A particular set of parameter settings 768 may correspond to a particular voice profile (e.g., whispered voice, excited voice, etc.).

[0131] In various embodiments of the present invention, speech model 722 and TTS front end 716 can be used to perform audio data synthesis based on model. TTS front end 716 can be identical with the front end used in traditional unit selection or parameter system. In other embodiments, some or all components of TTS front end 716 are based on other training models. However, the present invention is not limited to the TTS front end 716 of any particular type. Speech model 722 can be used for synthesized speech without the need for TTS unit storage device 772 or TTS parameter storage device 780, as described in more detail below.

[0132] The TTS front end 716 converts input text data 710 (from, for example, an application, a user, a device, or other text source) into a symbolic language representation, which may include language context features, such as phoneme data, punctuation data, syllable-level features, word-level features, and / or emotions, speakers, accents, or other features, for processing by the speech synthesis engine 718. Syllable-level features may include syllable emphasis, syllable speech rate, syllable inflection, or other such syllable-level features; word-level features may include word emphasis, word rate, word inflection, or other such word-level features. Emotional features may include data corresponding to emotions associated with the input text data 710, such as surprise, anger, or fear. Speaker features may include data corresponding to a speaker type, such as gender, age, or occupation. Accent features may include data corresponding to an accent associated with a speaker, such as a Southern accent, a Boston accent, an English accent, a French accent, or other such accents.

[0133] The TTS front end 716 can also process other input data 715, such as text tags or text metadata, which can indicate, for example, how a particular word is pronounced by indicating the desired output voice quality in a tag formatted according to Speech Synthesis Markup Language (SSML) or in some other form. For example, a first text tag can be included in the text, marking when the text should be whispered (e.g., <Begin Whisper>), and a second tag can be included in the text, marking when the text should be whispered (e.g., <End Whisper>). Tags can be included in the input text data 710 and / or the text for the TTS request can be accompanied by separate metadata indicating what text should be whispered (or having some other indicated audio characteristics). The speech synthesis engine 718 can compare the annotated speech unit model with the information stored in the TTS unit storage device 772 and / or the TTS parameter storage device 780 to convert the input text into speech. For example, the TTS front end 716 and the speech synthesis engine 718 can include their own controller / processor and memory, or they can use the controller / processor and memory of the server 120, the device 110 or other device. Similarly, instructions for operating the TTS front end 716 and the speech synthesis engine 718 may be located within the TTS component 780, within the server 120, within the memory and / or storage of the device 110, or within an external device.

[0134] The text data 710 input to the TTS component 780 can be sent to the TTS front end 716 for processing. The front end can include components for performing text normalization, language analysis, language prosody generation, or other such components. In the text normalization process, the TTS front end 716 can first process the text input and generate standard text, thereby converting numbers, abbreviations (such as Apt., St., etc.), symbols ($, %, etc.) into the equivalent of written words.

[0135] During language analysis, the TTS front end 716 can analyze the language in the normalized text to generate a sequence of speech units corresponding to the input text. This process can be called the conversion of graphemes to phonemes. The speech unit includes a symbolic representation of a sound unit, which will eventually be combined by the system and output as speech. For the purpose of speech synthesis, various sound units can be used to divide the text. The TTS component 780 can process speech based on phonemes (individual sounds), semiphones, diphones (the second half of a phoneme is coupled with the first half of an adjacent phoneme), diphones (two continuous phonemes), syllables, words, phrases, sentences or other units. Each word can be mapped to one or more speech units. This mapping can be performed using a language dictionary stored in the TTS storage component 772 using a system. The language analysis performed by the TTS front end 716 can also identify different grammatical components, such as prefixes, suffixes, phrases, punctuation, syntactic boundaries, etc. The TTS component 780 can use such grammatical components to produce an audio waveform output that sounds natural. The language dictionary may also include letter-to-sound rules and other tools that can be used to pronounce previously unrecognized words or letter combinations that may be encountered by the TTS component 780. Generally speaking, the more information contained in the language dictionary, the higher the quality of the speech output.

[0136] Based on the language analysis, the TTS front end 716 can then perform language prosody generation, in which speech units are annotated with desired prosodic features (also referred to as acoustic features), which indicate how the desired speech units will be pronounced in the final output speech. At this stage, the TTS front end 716 can take into account and combine any prosodic annotations accompanying the text input to the TTS component 780. Such acoustic features can include syllable-level features, word-level features, emotions, speakers, accents, languages, pitches, energy, durations, etc. The application of acoustic features can be based on the prosodic model available to the TTS component 780. This prosodic model indicates how a specific speech unit is pronounced in a specific situation. The prosodic model can take into account, for example, the position of a phoneme in a syllable, the position of a syllable in a word, the position of a word in a sentence or phrase, adjacent speech units, etc. Like a language dictionary, a prosodic model with more information may produce a higher quality speech output than a prosodic model with less information. In addition, prosodic models and / or speech units can be used to indicate specific voice qualities of the speech to be synthesized, where these voice qualities can match the voice qualities of the input speech (e.g., a speech unit can indicate prosodic features that should make the final synthesized speech sound like a whisper based on the input speech being whispered).

[0137] The output of the TTS front end 716, which may be referred to as a symbolic language representation, may include a sequence of speech units annotated with prosodic features. This symbolic language representation may be sent to a speech synthesis engine 718, which may also be referred to as a synthesizer, for converting into an audio waveform of speech to output to an audio output device and ultimately to a user. The speech synthesis engine 718 may be configured to convert input text into high-quality natural-sounding speech in an effective manner. This type of high-quality speech may be configured to sound as much as possible like a human speaker, or may be configured to be understandable to an audience without the need to attempt to imitate accurate human speech.

[0138] The speech synthesis engine 718 can perform speech synthesis using one or more different methods. In one synthesis method, referred to as unit selection, described further below, the unit selection engine 730 matches the symbolic language representation created by the TTS front end 716 with a database of recorded speech, such as a database (e.g., TTS unit storage device 772) that stores information about one or more speech corpora (e.g., speech lists 778a-n). Each speech list can correspond to individual audio clips recorded by a speaker (e.g., a voice actor), where these clips are stored in a separate list 778 as acoustic units (e.g., phonemes, bigrams, etc.). Each stored audio unit can also be associated with an index that lists various acoustic characteristics or other descriptive information about the unit. Each unit includes an audio waveform corresponding to the speech unit, such as a short .wav file of a specific sound, and a description of various features associated with the audio waveform. For example, the index entry for a particular unit can include, for example, the pitch, energy, duration, harmonics, center frequency of the particular unit, the position where the speech unit appears in a word, sentence or phrase, adjacent speech units, etc. The unit selection engine 730 may then use the information about each unit to select units to be connected together to form the speech output.

[0139] The unit selection engine 730 matches the symbolic language representation with the information about the spoken audio unit in the database. The unit database may include multiple examples of speech units to provide the system with many different options for connecting units into speech. Select the matching units determined to have the desired acoustic quality to create the desired output audio and connect them together (e.g., by the synthesis component 720) to form the output audio data 212 representing the synthesized speech. Using all the information in the unit database, the unit selection engine 730 can match the unit with the input text to select the unit that can form a natural sounding waveform. One benefit of unit selection is that, depending on the size of the database, a natural sounding speech output can be generated. As described above, the larger the unit database of the speech corpus, the more likely the system is to construct a natural sounding speech.

[0140] In another synthesis method (referred to as parametric synthesis), parameters such as frequency, volume, noise are changed by a parametric synthesis engine 732, a digital signal processor, or other audio generation device to create an artificial speech waveform output. Parametric synthesis uses a computerized speech generator, sometimes also referred to as a vocoder. Parametric synthesis can use acoustic models and various statistical techniques to match symbolic language representations with desired output speech parameters. Using parametric synthesis, a computing system (e.g., synthesis component 720) can generate an audio waveform with desired acoustic properties. Parametric synthesis may include the ability to be accurate at high processing speeds, and the ability to process speech without a large database associated with unit selection, but may also produce output speech quality that may not match the unit selection. Unit selection and parameter techniques may be performed alone or combined together and / or combined with other synthesis techniques to produce speech audio output.

[0141] The TTS component 780 may be configured to perform TTS processing in multiple languages. For each language, the TTS component 780 may include specially configured data, instructions, and / or components to synthesize speech in the desired language. To improve performance, the TTS component 780 may modify / update the contents of the TTS storage device 780 based on feedback of the TTS processing results, thereby enabling the TTS component 780 to improve speech recognition.

[0142] The TTS storage component 780 can be customized for a single user based on the personalized desired voice output of a single user. In particular, the voice unit stored in the unit database can be obtained from the input audio data of the user's speech. For example, in order to create the customized voice output of the system, the system can be configured with multiple voice lists 778a-278n, wherein each unit database is configured with different "voices" to match the required voice quality. Such voice lists can also be associated with user accounts. The voice for synthesized speech is selected by the TTS component 780. For example, a voice corpus can be stored for synthesizing whispered speech (or a voice approximating whispered speech), another voice corpus can be stored for synthesizing excited speech (or a voice approximating excited speech) and the like. In order to create different voice corpora, a large number of TTS training voices can be spoken by an individual (such as a voice actor) and recorded by the system. The audio associated with the TTS training speech can then be segmented into small audio clips and stored as a part of the voice corpus. The individual who speaks the TTS training utterances can speak with different voice qualities to create a customized voice corpus, for example, the individual can speak the training utterances in a whisper, speak the training utterances in an excited voice, etc. Thus, the audio of each customized voice corpus can be matched to the respective desired voice quality. The customized voice inventory 778 can then be used during runtime to perform unit selection to synthesize speech having a voice quality corresponding to the input voice quality.

[0143] In addition, parameter synthesis can be used to synthesize speech with desired voice quality. For parameter synthesis, parameter features that match the desired voice quality can be configured. If it is necessary to simulate an excited voice, the parameter features can indicate the increased speech rate and / or pitch of the produced voice. Many other examples are also possible. The desired parameter features of a specific voice quality can be stored in a "voice" profile (e.g., parameter settings 768) and used for speech synthesis when a specific voice quality is desired. Customized voices can be created based on a plurality of desired voice qualities (for unit selection or parameter synthesis) of a combination. For example, a voice can be "shouted", and another voice can be "shouted and reread". Many such combinations are possible.

[0144] Unit selection speech synthesis can be performed as follows. Unit selection includes a two-step process. First, the unit selection engine 730 determines what speech units to use, and then combines them so that a specific combined unit matches the required phonemes and acoustic features and creates the required speech output. The unit can be selected based on a cost function that represents the degree of match of a specific unit to the speech segment to be synthesized. The cost function can represent a combination of different costs that represent different aspects of how a specific speech unit works for a specific speech segment. For example, the target cost represents the degree of match of a single given speech unit to the characteristics of the desired speech output (e.g., pitch, prosody, etc.). The connection cost represents the degree of match of a specific speech unit to adjacent speech units (e.g., speech units that appear directly before or after a specific speech unit) in order to connect the speech units together in the final synthesized speech. The total cost function is a combination of the target cost, the connection cost, and other costs that can be determined by the unit selection engine 730. As part of unit selection, the unit selection engine 730 selects the speech unit with the lowest total combination cost. For example, if the connection cost is very high, it is not necessary to select a speech unit with a very low target cost.

[0145] The system can be configured with one or more speech corpora for unit selection. Each speech corpus can include a speech unit database. The speech unit database can be stored in the TTS unit storage device 772 or another storage component. For example, different unit selection databases can be stored in the TTS unit storage device 772. Each speech unit database (e.g., voice list) includes recorded speech utterances, wherein the corresponding text of the utterance is aligned with the utterance. The speech unit database may include many hours of recorded speech (in the form of audio waveforms, feature vectors or other formats), which may take up a lot of storage space. Unit samples in the speech unit database can be classified in a variety of ways, including by speech units (phonemes, bigrams, words, etc.), language prosodic tags, acoustic feature sequences, speaker identities, etc. Sample utterances can be used to create a mathematical model corresponding to the expected audio output of a specific speech unit. When matching symbolic language representations, the speech synthesis engine 718 can try to select the unit (including speech units and prosodic annotations) that best matches the input text in the speech unit database. Generally speaking, the larger the speech corpus / speech unit database, the better the speech synthesis will be, because a greater number of unit samples can be selected to form the exact desired speech output.

[0146] The parametric speech synthesis based on the vocoder can be performed as follows. The TTS component 780 may include an acoustic model or other model that can convert the symbolic language representation into a synthesized acoustic waveform for text input based on audio signal manipulation. The acoustic model includes rules that can be used by the parameter synthesis engine 732 to assign specific audio waveform parameters to input speech units and / or prosodic annotations. The rules can be used to calculate a score representing the likelihood that a specific audio output parameter (e.g., frequency, volume, etc.) corresponds to a portion of the input symbolic language representation from the TTS front end 716.

[0147] The parameter synthesis engine 732 can use a variety of techniques to match the speech to be synthesized with the input speech units and / or prosodic annotations. A common technique is to use a hidden Markov model (HMM). HMM can be used to determine the probability that the audio output should match the text input. HMM can be used to convert parameters from the language and acoustic space into parameters used by a vocoder (digital speech encoder) to artificially synthesize the desired speech. Using HMM, many states are presented, where the states together represent one or more potential acoustic parameters to be output to the vocoder, and each state is associated with a model, such as a Gaussian mixture model. The transition between states may also have an associated probability, indicating the possibility of reaching the current state from the previous state. The sound to be output can be represented as a path between HMM states, and multiple paths can represent multiple possible audio matches of the same input text. Each part of the text can be represented by multiple potential states corresponding to different known phoneme pronunciations and parts thereof (such as phoneme identifiers, stress, accents, positions, etc.). The initial determination of the probability of a potential phoneme can be associated with a state. When the speech synthesis engine 718 processes a new text, the state can change or remain unchanged based on the processing of the new text. For example, the pronunciation of a previously processed word may change based on a later processed word. The Viterbi algorithm can be used to find the most likely sequence of states based on the processed text. The HMM can generate speech in a parameterized form, including parameters such as fundamental frequency (f0), noise envelope, spectrum envelope, etc., which are converted into audio clips by the vocoder. The output parameters can be configured for a specific vocoder, such as a STRAIGHT vocoder, a TANDEM-STRAIGHT vocoder, a WORLD vocoder, a HNM (harmonic plus noise) based vocoder, a CELP (code excited linear prediction) vocoder, a GlottHMM vocoder, an HSM (harmonic / stochastic model) vocoder, or other vocoders.

[0148] In addition to computing the latent state of an audio waveform as a potential match to a speech unit, the parametric synthesis engine 732 can also compute the latent states of other potential audio outputs (e.g., various ways of uttering a particular phoneme or bigram) as potential acoustic matches to an acoustic unit. In this way, multiple states and state transition probabilities can be computed.

[0149] The possible states and possible state transitions calculated by the parameter synthesis engine 732 can result in multiple potential audio output sequences. Based on the acoustic model and other potential models, the potential audio output sequence can be scored according to the confidence level of the parameter synthesis engine 732. The highest scoring audio output sequence including the parameter stream to be synthesized can be selected, and digital signal processing can be performed by a vocoder or similar component to create an audio output, which includes a synthesized speech waveform corresponding to the parameters of the highest scoring audio output sequence and, if the correct sequence is selected, also corresponding to the input text. Different parameter settings 768 that can represent acoustic settings that match a specific parameter "voice" can be used by the synthesis component 722 to ultimately create output audio data 212.

[0150] When performing unit selection, after the unit selection engine 730 selects a unit, the audio data corresponding to the unit can be passed to the audio data modification component 720. The audio data modification component 720 can then process the audio data of the unit to create modified audio data, wherein the modified audio data reflects the desired audio quality. The audio data modification component 720 can store various operations that can convert the unit audio data into the modified audio data, wherein different operations can be performed based on the desired audio effect (e.g., whispering, shouting, etc.).

[0151] For example, input text can be received together with metadata (e.g., SSML tags) indicating that the selected portion of the input text should be whispered when output by the TTS module 780. For each unit corresponding to the selected portion, the audio data modification component 720 can process the audio data of the unit to create the modified unit audio data. The modified unit audio data can then be connected to form the output audio data 212. The modified unit audio data can also be connected with the unmodified audio data, depending on when the desired whispered voice begins and / or ends. Although the modified audio data may be sufficient to make the output audio data have the desired audio quality, other factors may also affect the final output of the audio, such as playback speed, background effects, etc., which may be beyond the control range of the TTS module 780. In that case, other output data 785 can be output together with the output audio data 212 so that the final playback device (e.g., device 110) receives playback instructions that can help create the desired output audio. Therefore, other output data 785 can include instructions or other data indicating playback device settings (e.g., volume, playback rate, etc.), or indicate how to output other data of the output audio data including the synthesized speech. For example, for whispered speech, the output audio data 212 may include other output data 785, which may include a prosody tag or other indicator that instructs the device 110 to slow down the playback of the output audio data 212, thereby producing a final audio sound that is more like whispered speech, generally slower than normal speech. In another example, the other output data 785 may include a volume tag that instructs the device 110 to output the speech at a volume level that is less than the current volume setting of the device 110, thereby improving the quiet whisper effect.

[0152] like Figure 8 As shown, the user recognition component 295 may include one or more subcomponents, including a visual component 808, an audio component 810, a biometric component 812, a radio frequency (RF) component 814, a machine learning (ML) component 816, and a recognition confidence component 818. In some cases, the user recognition component 295 may monitor data from one or more subcomponents and determine to determine the identity of one or more users associated with the data input to the system. The user recognition component 295 may output user recognition data 895, which may include a user identifier associated with the user that the system believes is initiating the input of data to the system. As described below, the user recognition data 895 may be used to inform the process performed by the coordinator 240 (or its subcomponents).

[0153] The visual component 808 can receive data from one or more sensors (e.g., cameras) that can provide images or sensors (e.g., motion sensors) that indicate motion. The visual component 808 can perform facial recognition or image analysis to determine the identity of the user and associate the identity with the user profile associated with the user. In some cases, when the user faces the camera, the visual component 808 can perform facial recognition and identify the user with a high degree of confidence. In other cases, the visual component 808 may have a low confidence in the identity of the user, and the user recognition component 295 can utilize the determination from the additional components to determine the identity of the user. The visual component 808 can be used in combination with other components to determine the identity of the user. For example, the user recognition component 295 can use the data from the visual component 808 and the data from the audio component 810 to identify that the user's face looks like speaking while the user device 110 captures audio, for identifying the user who speaks input to the user device 110.

[0154] User device 110 may include a biometric sensor that transmits data to biometric component 812. For example, biometric component 812 may receive data corresponding to a fingerprint, an iris or retina scan, a thermal scan, the user's weight, the user's body shape, pressure (e.g., within a floor sensor), etc., and may determine a biometric profile corresponding to the user. For example, biometric component 812 may distinguish between a user and a voice from a television. Thus, biometric component 812 may incorporate biometric information into a confidence level to determine the identity of the user.

[0155] RF component 814 can use RF positioning to track a device that a user may carry or wear. For example, a user may be associated with a device. The device may transmit RF signals (e.g., Wi-Fi, , etc.). User device 110 may detect the signal and indicate the strength of the signal to RF component 814 (e.g., as a received signal strength indication (RSSI)). RF component 814 may use the RSSI to determine the identity of the user (with an associated confidence level). In some cases, RF component 814 may determine that the received RF signal is associated with a mobile device associated with a particular user identifier.

[0156] In some cases, user device 110 may include some RF or other detection processing capabilities so that the user speaking the input can scan, click, or otherwise identify his / her personal device to user device 110. In this way, the user can "register" with user device 110 so that user device 110 can determine who spoke a particular input. This registration can occur before, during, or after the input is spoken.

[0157] The ML component 816 can track various user behaviors as a factor in determining the confidence level of the user's identity. For example, a user may follow a regular schedule that places the user in a first location during the day (e.g., at work or at school). In this example, the ML component 816 factors in past behaviors and / or trends to determine the identity of the user providing input to the user device 110. Thus, the ML component 816 can use historical data and / or usage patterns over time to increase or decrease the confidence level of the user's identity.

[0158] In some cases, the recognition confidence component 818 receives determinations from the various components 808 , 810 , 812 , 814 , and 816 and can determine a final confidence level associated with the identity of the user. The confidence level or other scoring data can be included in the user recognition data 895 .

[0159] The audio component 810 may receive data from one or more sensors (e.g., one or more microphones) capable of providing an audio signal to facilitate identifying a user. The audio component 810 may perform audio recognition on the audio signal to determine the identity of the user and an associated user identifier. In some cases, the audio component 810 may perform voice recognition to determine the identity of the user.

[0160] The audio component 810 may also perform user identification based on the audio received by the user device 110. The audio component 810 may determine a score indicating whether speech in the audio originates from a particular user. For example, a first score may indicate a likelihood that speech in the audio originates from a first user associated with a first user identifier, a second score may indicate a likelihood that speech in the audio originates from a second user associated with a second user identifier, and so on. The audio component 810 may perform user identification by comparing audio features representing the audio with stored audio features of the user.

[0161] Fig. 9 is a block diagram conceptually illustrating a user device 110 that may be used with the system. Fig.10is a block diagram conceptually illustrating example components of a remote device, such as system 120 and skill system 225 that can assist in ASR processing, NLU processing, etc. The term "server" as used herein may refer to a traditional server as understood in a server / client computing structure, but may also refer to a plurality of different computing components that can assist in the operations discussed herein. For example, a server may include one or more physical computing components (e.g., a rack server) that are physically and / or network-connected to other devices / components and are capable of performing computing operations. A server may also include one or more virtual machines that simulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, etc. to perform the operations discussed herein. A server may be configured to operate using one or more of a client-server model, a computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, a peer-to-peer model, sandbox technology, or other computing technologies.

[0162] The system may include multiple servers (120 / 225), such as one or more servers 120 for performing ASR processing, one or more servers 120 for performing NLU processing, one or more skill systems 225 for performing actions in response to user input, etc. In operation, each of these devices (or groups of devices) may include computer-readable and computer-executable instructions resident on the respective device (120 / 225), as will be discussed further below.

[0163] Each of these devices (110 / 120 / 225) may include one or more controllers / processors (904 / 1004), which may each include a central processing unit (CPU) for processing data and computer-readable instructions and a memory (906 / 1006) for storing data and instructions for the corresponding device. The memory (906 / 1006) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (110 / 120 / 225) may also include a data storage component (908 / 1008) for storing data and controller / processor executable instructions. Each data storage component (908 / 1008) may individually include one or more non-volatile storage device types, such as magnetic storage devices, optical storage devices, solid-state storage devices, etc. Each device (110 / 120 / 225) may also be connected to a removable or external non-volatile memory and / or storage device (eg, a removable memory card, a memory key drive, a network storage device, etc.) via a corresponding input / output device interface (902 / 1002).

[0164] Computer instructions for operating each device (110 / 120 / 225) and its various components may be executed by the respective device controller / processor (904 / 1004) at runtime using the memory (906 / 1006) as a temporary "working" storage device. The computer instructions for a device may be stored in a non-volatile memory (906 / 1006), storage device (908 / 1008), or an external device in a non-transitory manner. Alternatively, in addition to or in lieu of software, some or all of the executable instructions may be embedded in hardware or firmware on the respective device.

[0165] Each device (110 / 120 / 225) includes an input / output device interface (902 / 1002). Various components may be connected via the input / output device interface (902 / 1002), as will be discussed further below. In addition, each device (110 / 120 / 225) may include an address / data bus (924 / 1024) for transferring data between components of the respective device. In addition to (or in lieu of) connecting to other components across the bus (924 / 1024), each component within the device (110 / 120 / 225) may also be directly connected to other components.

[0166] refer to Fig. 9 , the user device 110 may include an input / output device interface 902, which is connected to various components, such as an audio output component, such as a speaker 912, a wired headset or a wireless headset (not shown), or other components capable of outputting audio. The user device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 920 or a microphone array, a wired headset or a wireless headset (not shown), etc. If a microphone array is included, the approximate distance to the origin point of the sound can be determined by acoustic positioning based on the time and amplitude differences between the sounds captured by different microphones in the array. The user device 110 may additionally include a display 916 for displaying content. The user device 110 may also include a camera 918.

[0167] Via antenna 914, input / output device interface 902 may be connected to one or more networks 199 via a wireless local area network (WLAN) (e.g., Wi-Fi) radio, Bluetooth, and / or a wireless network radio, such as a radio capable of communicating with a wireless communication network such as a Long Term Evolution (LTE) network, a WiMAX network, a 3G network, a 4G network, a 5G network, etc. Wired connections such as Ethernet may also be supported. Through network 199, the system may be distributed in a network environment. The I / O device interface (902 / 1002) may also include a communication component that allows data to be exchanged between devices (e.g., different physical servers or other components in a collection of servers).

[0168] Components of device 110, system 120, or skill system 225 may include their own dedicated processors, memory, and / or storage. Alternatively, one or more of the components of device 110, system 120, or skill system 225 may utilize the I / O interface (902 / 1002), processor (904 / 1004), memory (906 / 1006), and / or storage (908 / 1008) of device 110, system 120, or skill system 225, respectively. Thus, ASR component 250 may have its own I / O interface, processor, memory, and / or storage; NLU component 260 may have its own I / O interface, processor, memory, and / or storage; and so on for the various components discussed herein.

[0169] As described above, multiple devices may be employed in a single system. In such a multi-device system, each of the devices may include different components for performing different aspects of the system processing. Multiple devices may include overlapping components. As described herein, the components of user device 110, system 120, and skill system 225 are illustrative and may be located as independent devices or may be included in whole or in part as components of a larger device or system.

[0170] like Fig.11 As shown, multiple devices (110a-110g, 120, 225) may include components of the system, and the devices may be connected via a network 199. The network 199 may include a local or private network, or may include a wide area network (e.g., the Internet). The devices may be connected to the network 199 via a wired or wireless connection. For example, a vehicle 110a, a smartphone 110b, a smart watch 110c, a tablet computer 110d, a vehicle 110e, a display vehicle 110f, and / or a smart TV 110g may be connected to the network 199 via a wireless service provider, via a Wi-Fi or cellular network connection, etc. Other devices are included as supporting devices for connecting to the network, such as the system 120, the skill system 225, and / or other devices. The supporting devices may be connected to the network 199 via a wired connection or a wireless connection. The network device may capture audio using one or more built-in or connected microphones or other audio capture devices, where processing is performed by an ASR component, NLU component, or other component of the same device or another device connected via network 199 (e.g., ASR component 250, NLU component 260, etc. of one or more servers 120).

[0171] The foregoing may also be understood in light of the following terms:

[0172] 1. A method for processing data representing a spoken command, the method comprising:

[0173] receiving first audio data representing speech from a voice control device;

[0174] Determining the first audio data includes:

[0175] a first portion of audio data representing a wake-up word corresponding to a first speech processing system having a first style of synthesized speech, and

[0176] a second portion of the audio data, the second portion representing a command;

[0177] determining a domain associated with the command;

[0178] determining that the domain is associated with a second speech processing system having a second style of synthesized speech;

[0179] sending command data including the command from the first speech processing system to the second speech processing system;

[0180] receiving a first response to the command from the second speech processing system, the first response comprising a representation in a first natural language corresponding to the second style of synthesized speech;

[0181] determining a second response to the command using the first speech processing system and the first response, the second response comprising a representation in a second natural language corresponding to the first style of synthesized speech; and

[0182] Second audio data corresponding to the second response is sent to the voice control device.

[0183] 2. The method according to clause 1, further comprising:

[0184] determining that the command corresponds to a set of actions that occur later; and

[0185] sending a second command to execute the action from the second speech processing system to the voice control device,

[0186] Wherein the second audio data comprises a representation of a name of the second speech processing system.

[0187] 3. The method according to clause 1 or 2, further comprising:

[0188] receiving third audio data representing a second utterance from the voice control device;

[0189] determining that the third audio data corresponds to a second command;

[0190] determining that the second command corresponds to a user device in proximity to the user; and

[0191] An instruction corresponding to the second command is sent to the user device via an application program interface.

[0192] 4. A computer-implemented method, the computer-implemented method comprising:

[0193] receiving input data corresponding to the command;

[0194] determining, using the first natural understanding system, that the command is associated with a second natural understanding system;

[0195] sending first data corresponding to the command from the first natural understanding system to the second natural understanding system;

[0196] receiving, from the second natural understanding system, second data corresponding to a first response to the command, the second data comprising a first indication of the second natural understanding system;

[0197] determining, using the first natural understanding system and the second data, third data corresponding to a second response to the command, the third data including a second indication of the first natural understanding system; and

[0198] An output corresponding to the third data is generated.

[0199] 5. The computer-implemented method of clause 4, further comprising:

[0200] Prior to generating the output, determining that the second nature understanding system is associated with a permission request;

[0201] Before sending the first data, determining to generate a second output corresponding to the permission request; and

[0202] receiving fourth input data corresponding to the permission request,

[0203] Wherein the third data further comprises a representation of the permission request.

[0204] 6. The computer-implemented method of clause 4 or 5, further comprising:

[0205] determining that the command corresponds to a second output of the second natural understanding system, the second output occurring after the output,

[0206] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0207] 7. The computer-implemented method of clause 4, 5 or 6, further comprising:

[0208] determining that receiving the input data is associated with a first device; and

[0209] determining that generating the output is associated with a second device,

[0210] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0211] 8. The computer-implemented method of clause 4, 5, 6 or 7, further comprising:

[0212] determining that receiving the input data is associated with a first device;

[0213] determining a user account associated with the first device; and

[0214] determining that the user account includes a third indication of the second nature understanding system,

[0215] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0216] 9. The computer-implemented method of clause 4, 5, 6, 7 or 8, further comprising:

[0217] A determination is made that the input data includes a representation of a wake word associated with the first natural understanding system.

[0218] 10. The computer-implemented method of clause 4, 5, 6, 7, 8, or 9, wherein determining that the command is associated with the second natural understanding system comprises:

[0219] determining a domain corresponding to the input data; and

[0220] It is determined that the second nature understanding system corresponds to the domain.

[0221] 11. The computer-implemented method of clause 4, 5, 6, 7, 8, 9 or 10, further comprising:

[0222] receiving second audio data corresponding to a second command;

[0223] determining that the second command is associated with the second nature understanding system; and

[0224] A third command to process third audio data is sent to the second natural understanding system.

[0225] 12. The computer-implemented method of clause 4, 5, 6, 7, 8, 9, 10, or 11, further comprising:

[0226] determining that the second data lacks the first information;

[0227] transmitting, from the first nature understanding system to the second nature understanding system, fourth data corresponding to the request for the first information; and

[0228] Fifth data corresponding to the first information is received from the second nature understanding system.

[0229] 13. A computing device, comprising:

[0230] at least one processor; and

[0231] at least one memory comprising instructions that, when executed by the at least one processor, cause the computing device to:

[0232] receiving input data corresponding to the command;

[0233] determining, using the first natural understanding system, that the command is associated with a second natural understanding system;

[0234] sending first data corresponding to the command from the first natural understanding system to the second natural understanding system;

[0235] receiving, from the second natural understanding system, second data corresponding to a first response to the command, the second data comprising a first indication of the second natural understanding system;

[0236] determining, using the first natural understanding system and the second data, third data corresponding to a second response to the command, the third data including a second indication of the first natural understanding system; and

[0237] An output corresponding to the third data is generated.

[0238] 14. The computing device of clause 13, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

[0239] Prior to generating the output, determining that the second nature understanding system is associated with a permission request;

[0240] Before sending the first data, determining to generate a second output corresponding to the permission request; and

[0241] receiving fourth input data corresponding to the permission request,

[0242] Wherein the third data further comprises a representation of the permission request.

[0243] 15. The computing device of clause 13 or 14, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

[0244] determining that the command corresponds to a second output of the second natural understanding system, the second output occurring after the output,

[0245] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0246] 16. The computing device of clause 13, 14 or 15, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

[0247] determining that receiving the input data is associated with a first device; and

[0248] determining that generating the output is associated with a second device,

[0249] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0250] 17. The computing device of clause 13, 14, 15 or 16, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

[0251] determining that receiving the input data is associated with a first device; and

[0252] determining a user account associated with the first device; and

[0253] determining that the user account includes a third indication of the second nature understanding system,

[0254] Wherein the third data comprises a representation of a name of the second nature understanding system.

[0255] 18. A computing device as described in clause 13, 14, 15, 16 or 17, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

[0256] A determination is made that the input data includes a representation of a wake word associated with the first natural understanding system.

[0257] 19. The computing device of clause 13, 14, 15, 16, 17, or 18, wherein the at least one memory further comprises instructions for determining that the command is associated with the second natural understanding system and which, when executed by the at least one processor, further cause the computing device to:

[0258] determining a domain corresponding to the input data; and

[0259] It is determined that the second nature understanding system corresponds to the domain.

[0260] 20. The computing device of clause 13, 14, 15, 16, 17, 18 or 19, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

[0261] determining that the second data lacks the first information;

[0262] transmitting, from the first nature understanding system to the second nature understanding system, fourth data corresponding to the request for the first information; and

[0263] Fifth data corresponding to the first information is received from the second nature understanding system.

[0264] The concepts disclosed herein may be applied in many different devices and computer systems, including, for example, general purpose computing systems, speech processing systems, and distributed computing environments.

[0265] The above aspects of the present disclosure are intended to be illustrative. The aspects are selected to illustrate the principles and applications of the present disclosure, and are not intended to be exhaustive or to limit the present disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those of ordinary skill in the field of computer and speech processing will recognize that the components and processing steps described herein may be interchangeable with other components or steps or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. In addition, those skilled in the art will appreciate that the present disclosure may be practiced without some or all of the specific details and steps disclosed herein.

[0266] Aspects of the disclosed system may be implemented as a computer method or article of manufacture, such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be readable by a computer and may include instructions for causing a computer or other device to perform the processes described in the present disclosure. The computer-readable storage medium may be implemented via volatile computer memory, non-volatile computer memory, a hard drive, a solid-state memory, a flash drive, a removable disk, and / or other media. In addition, components of the system may be implemented as firmware or hardware, such as an acoustic front end (AFE), which includes, among other things, analog and / or digital filters (e.g., filters configured as firmware of a digital signal processor (DSP)).

[0267] Unless otherwise specifically stated, or understood otherwise in the context of use, conditional language used herein, such as "can," "may," "might," "could," "for example," and the like, is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements, or steps. Thus, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included or to be performed in any particular embodiment with or without other input or prompting. The terms "comprising," "including," "having," and the like are synonymous and are used in an open form as inclusive and do not exclude additional elements, features, actions, operations, and the like. In addition, the term "or" is used in its inclusive sense (rather than an exclusive sense) such that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

[0268] Unless otherwise specifically stated, negative conjunction language such as the phrase "at least one of X, Y, Z" should be understood in the context as generally used to indicate that an item, term, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such negative conjunction language is generally not intended to and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0269] As used in this disclosure, the terms "a" or "an" may include one or more items unless specifically stated otherwise. In addition, the phrase "based on" is intended to mean "based at least in part on" unless specifically stated otherwise.

Claims

1. A computer-implemented method, the computer-implemented method comprising: receiving input data corresponding to the command; determining, using the first natural understanding system, that the command is associated with a second natural understanding system; sending first data corresponding to the command from the first natural understanding system to the second natural understanding system; receiving, from the second natural understanding system, second data corresponding to a first response to the command, the second data comprising a first indication of the second natural understanding system; determining, using the first natural understanding system and the second data, third data corresponding to a second response to the command, the third data including a second indication of the first natural understanding system; as well as An output corresponding to the third data is generated.

2. The computer-implemented method of claim 1 , further comprising: Prior to generating the output, determining that the second nature understanding system is associated with a permission request; Before sending the first data, determining to generate a second output corresponding to the permission request; as well as receiving fourth input data corresponding to the permission request, Wherein the third data further comprises a representation of the permission request.

3. The computer-implemented method of claim 1 , further comprising: determining that the command corresponds to a second output of the second natural understanding system, the second output occurring after the output, Wherein the third data comprises a representation of a name of the second nature understanding system.

4. The computer-implemented method of claim 1 , further comprising: determining that receiving the input data is associated with a first device; as well as determining that generating the output is associated with a second device, Wherein the third data comprises a representation of a name of the second nature understanding system.

5. The computer-implemented method of claim 1 , further comprising: determining that receiving the input data is associated with a first device; determining a user account associated with the first device; as well as determining that the user account includes a third indication of the second nature understanding system, Wherein the third data comprises a representation of a name of the second nature understanding system.

6. The computer-implemented method of claim 1 , further comprising: A determination is made that the input data includes a representation of a wake word associated with the first natural understanding system.

7. The computer-implemented method of any one of claims 1 to 6, wherein determining that the command is associated with the second natural understanding system comprises: determining a domain corresponding to the input data; as well as It is determined that the second nature understanding system corresponds to the domain.

8. The computer-implemented method of any one of claims 1 to 6, further comprising: receiving second audio data corresponding to a second command; determining that the second command is associated with the second natural understanding system; as well as A third command to process third audio data is sent to the second natural understanding system.

9. The computer-implemented method of any one of claims 1 to 6, further comprising: determining that the second data lacks the first information; sending, from the first nature understanding system to the second nature understanding system, fourth data corresponding to the request for the first information; as well as Fifth data corresponding to the first information is received from the second nature understanding system.

10. A computing device, comprising: at least one processor; as well as at least one memory comprising instructions that, when executed by the at least one processor, cause the computing device to: receiving input data corresponding to the command; determining, using the first natural understanding system, that the command is associated with a second natural understanding system; sending first data corresponding to the command from the first natural understanding system to the second natural understanding system; receiving, from the second natural understanding system, second data corresponding to a first response to the command, the second data comprising a first indication of the second natural understanding system; determining, using the first natural understanding system and the second data, third data corresponding to a second response to the command, the third data including a second indication of the first natural understanding system; as well as An output corresponding to the third data is generated.

11. The computing device of claim 10, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to: prior to generating the output, determining that the second nature understanding system is associated with a permission request, Before sending the first data, determining to generate a second output corresponding to the permission request; and receiving fourth input data corresponding to the permission request, Wherein the third data further comprises a representation of the permission request.

12. The computing device of claim 10, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to: determining that the command corresponds to a second output of the second natural understanding system, the second output occurring after the output, Wherein the third data comprises a representation of a name of the second nature understanding system.

13. The computing device of any one of claims 10 to 12, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to: determining that receiving the input data is associated with a first device; and determining that generating the output is associated with a second device, Wherein the third data comprises a representation of a name of the second nature understanding system.

14. The computing device of any one of claims 10 to 12, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to: determining that receiving the input data is associated with a first device; and determining a user account associated with the first device; and determining that the user account includes a third indication of the second nature understanding system, Wherein the third data comprises a representation of a name of the second nature understanding system.

15. The computing device of any one of claims 10 to 12, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to: A determination is made that the input data includes a representation of a wake word associated with the first natural understanding system.

Images