Method for modifying a set of one or more wake-up words of a voice-enabled device

Abstract

The present disclosure relates to dynamic wake-up words for voice-enabled devices. A system and method are disclosed that can parse a spoken utterance into a natural language request and a speech audio segment, wherein the natural language request instructs the system to use the speech audio segment as a new wake-up word. In response to this wake-up word assignment instruction, the system and method are further capable of immediately building a new wake-up word detector to activate the device when the new wake-up word in the input audio is matched. Different methods for quickly building a new wake-up word detector are described. Variations of the wake-up word assignment instructions can make the new wake-up word public or private. They can also add the new wake-up word to an earlier wake-up word, or replace the earlier wake-up word.

Description

Technical Field

[0001] The present technology relates to wake-up words for voice-enabled devices, and in particular to assigning dynamic wake-up words using natural language queries and instantly building a wake-up word spotter for the wake-up word for one or more of a user's voice-enabled devices. Background Art

[0002] Automatic speech recognition (ASR) systems that recognize human speech, along with natural language understanding (NLU) capabilities that extract the meaning of speech, offer enormous potential as a simple and natural way to interact with speech-enabled devices. Such systems are enabled, in part, by the vast computing and communication resources available in modern devices. Advanced speech understanding systems, such as virtual assistants, have been developed that can recognize a wide variety of speech and handle complex requests in different languages and dialects.

[0003] Virtual assistants do not respond to verbal requests when idle. They wake up or activate when they receive an activation signal, such as a tap, a button press, or a verbal activation phrase called a wake word (or wake phrase), and switch their state from idle to active. The use of a wake word is key to hands-free and eyes-free operation of voice-enabled devices. In the active state, virtual assistants respond to user requests. They typically return to the idle state after responding to the request. When idle, voice-enabled devices continuously monitor incoming audio to detect the wake word. To reduce power consumption, some devices can operate in a low-power mode when the virtual assistant is idle; when activated, they return to full-power mode.

[0004] A wake word is typically a word or a short phrase. A continuously operating module that monitors incoming audio to detect the wake word is called a wake word detector. Various commercial implementations of wake words for voice-enabled devices include "Hey, Siri," "OK, Google," and "Alexa." Voice-enabled devices can be sold with both a factory-installed wake word and a wake word detector ready to detect a predefined wake word.

[0005] A wake word detector is an audio processing algorithm specifically designed to detect a designated wake word or set of designated wake words in a continuous audio stream. The algorithm typically runs continuously at a fixed frame rate and must be highly efficient. On devices in low-power mode, the detector can run continuously without drawing excessive power, thereby conserving battery life.

[0006] Sometimes it may be desirable to customize the factory-installed wake-up word on one or more voice-enabled devices. For example, in a home or office environment, there may be multiple devices that use the same factory-installed wake-up word as the activation phrase. This can result in erroneous device activations, or conflicts where multiple devices activate when a common wake-up word is sensed. Multiple device activations can cause various problems depending on the type of request that follows the wake-up word. For example, a request to play music can result in multiple devices playing the same song (out of sync) or different songs at the same time. A request to send a message can result in multiple copies of the message being sent. These and other conflict scenarios lead to a poor user experience.

[0007] A key challenge when providing dynamic wake-up words is training new wake-up word detectors in a very short time. Factory-installed wake-up word detectors are typically trained using large datasets of audio samples that include positive instances specifically recorded for one or more given wake-up words, and may include some negative instances. Such labeled samples are used to train a classifier algorithm, such as a recurrent neural network, to distinguish a given wake-up word (or multiple wake-up words) from non-wake-up word speech in an audio stream. Unfortunately, traditional methods for collecting audio sample data are not available for dynamic wake-up words, which require building a detector immediately without having to collect a large dataset of audio samples for the dynamic wake-up words. Summary of the Invention

[0008] According to one aspect of the present disclosure, a method for modifying a set of one or more wake-up words for a voice-enabled device is provided. The method includes: receiving a verbal request from a user; parsing the verbal request into a natural language request and a speech audio segment, wherein the natural language request indicates that the device accepts the speech audio segment as a new wake-up word; and constructing a new wake-up word detector to identify the new wake-up word as an activation trigger.

[0009] According to another aspect of the present disclosure, a method for modifying a set of one or more wake-up words for a voice-enabled device is provided. The method includes: accepting a spoken utterance; parsing the utterance into a natural language request and a speech audio segment, wherein the natural language request indicates that the device accepts the speech audio segment as a new wake-up word; using automatic speech recognition to map the new wake-up word to a new wake-up word phoneme sequence; and constructing a new wake-up word detector to recognize the new wake-up word phoneme sequence as an activation trigger by the following steps: dividing the new wake-up word phoneme sequence into a sequence of two or more consecutive partial phoneme segments; for each partial phoneme segment, providing a corresponding partial wake-up word detector; and sequentially assembling the provided partial wake-up word detectors into the new wake-up word detector for the entire new wake-up word phoneme sequence.

[0010] According to another aspect of the present disclosure, a method for modifying a set of one or more wake-up words for a voice-enabled device is provided. The method includes: receiving a spoken request; parsing the spoken request into a natural language request and a speech audio segment, wherein the natural language request indicates that the device accepts the speech audio segment as a new wake-up word; and defining a new wake-up word detector to identify the new wake-up word as an activation trigger by the following steps: determining additional speech audio samples of the speech audio segment, converting the speech audio segment and the additional speech audio samples into a phoneme sequence, and defining the new wake-up word detector based on one or more phoneme sequences in the phoneme sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic representation of a voice-enabled device in accordance with an embodiment of the present technology.

[0012] Figure 2 is a schematic representation of a server according to an embodiment of the present technology.

[0013] Figure 3 is a flow chart illustrating the operation of a voice-enabled device and a server in processing a user query according to an embodiment of the present technology.

[0014] Figure 4 is a flow diagram of the operation of a system for making a wake word public, private, additive, or exclusive.

[0015] Figure 5 is a flowchart providing further details of an embodiment of a detector builder based on a wake-up word sampling method.

[0016] Figure 6 is a flow chart providing further details of an embodiment of a detector builder based on a continuous transcription approach.

[0017] Figure 7 is a flow chart that provides further details of an embodiment of a detector builder based on a wake-word segmentation approach.

[0018] Figure 8 is a diagram of a wake-up word segmentation method for processing dynamic wake-up words.

[0019] Figure 9 is a schematic block diagram of a computing environment according to an embodiment of the present technology. Specific embodiments

[0020] Automatic speech recognition (ASR) systems that recognize human speech, along with natural language understanding (NLU) capabilities that extract the meaning of speech, offer enormous potential as a simple and natural way to interact with voice-enabled devices. Such systems are enabled, in part, by the vast computing and communication resources available in modern devices. Advanced speech understanding systems have been developed that can process complex utterances to recognize a wide variety of speech in different languages and dialects.

[0021] The present technology will now be described with reference to the figures, which, in an embodiment, relate to a system capable of parsing received utterances into natural language requests and speech audio segments, wherein the request instructs the system to use the speech audio segment as a new wake-up word. This type of request will be referred to as a wake-up word assignment instruction (WAD). In response to such a request, the system is further capable of building a new wake-up word detector to identify the new wake-up word, and the construction of the detector is fast enough that the new wake-up word can be used immediately after the system responds to the WAD.

[0022] In the context of voice-enabled systems, the terms utterance, query, and request are closely related and are sometimes used interchangeably. A spoken natural language request from a user is conveyed both as voice audio (utterance) and (if correctly transcribed) as words (query). The device can perform various actions in response to a general query.

[0023] A wake word assignment command, or command or WAD for short, is a request to a device to change its set of wake words by adding or replacing them.

[0024] One such action may be a natural language request to assign a new wake-up word in the form of a wake-up word assignment instruction. Upon recognizing such a wake-up word assignment instruction, the present technology may immediately build a new wake-up word detector for the dynamic wake-up word. As used in this context, the term "immediately" means that the new wake-up word detector may be built within seconds of receiving the new wake-up word, as described in more detail below. Without the aid of a large dataset of audio instances of the new wake-up word, the use of dynamic wake-up words requires other methods to quickly build a new wake-up word detector. These methods include at least the following three and their variations:

[0025] 1. Wake-up word segmentation method;

[0026] 2. Wake-up word sampling method; and

[0027] 3. Continuous transcription method.

[0028] Each of these methods is described in detail below. Immediately after the construction of the new wake word detector is completed (eg, within a few seconds), the dynamic wake word and its detector can be stored and ready to activate the device.

[0029] The parser can further identify optional parameters that define variations of the instruction as part of the predefined wake word assignment instruction template; these parameters can be attributes of the dynamic wake word (i.e., how it is identified) or attributes of the instruction (i.e., how the WAD is fulfilled). For example, a user can specify that the new wake word is public, meaning that other users can use the same dynamic wake word to wake the device, or specify that the new wake word is private, meaning that the new wake word is valid only for that user and excludes others. As another example, a user can specify whether the new wake word will replace an earlier wake word or will be used in addition to the earlier wake word.

[0030] Instruction parameters have natural language wording that conveys the specific parameter values that can be found in the instruction. When an optional parameter is missing, the parameter can have an implicit default value related to the implementation, or the system can prompt the user to enter a value. In this disclosure, optional instruction parameters will be simply referred to as parameters.

[0031] It should be understood that the present invention can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete and the present invention will be fully conveyed to those skilled in the art. In fact, the present invention is intended to encompass alternatives, modifications, and equivalents within the scope and spirit of the present invention as defined by the appended claims that these embodiments are included. In addition, in the following detailed description of the present invention, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be clear to those skilled in the art that the present invention can be put into practice without these specific details.

[0032] Figure 1 is a schematic block diagram of an example voice-enabled device 100 within which the present technology may be implemented. Device 100 may be or include an agent having any of a variety of electronic or electromechanical components configured to accept voice requests, including, for example, cellular phones, digital assistants, tablets and other computing devices, automobile control systems, and other devices.

[0033] Reference below Figure 9 A more detailed description of an example voice-enabled device 100 is described, but generally, the device 100 may include a processor 102 configured to control operations within the device 100 and to facilitate communications between various components within the device 100. The processor 102 may include a standardized processor, a dedicated processor, a microprocessor, etc. that may execute instructions to control the device 100.

[0034] The processor 102 can receive and process input from various input devices, including one or more microphones 106. The microphone 106 can include a transducer or sensor that can receive sound and convert it into an electrical signal. According to one embodiment, the microphone 106 can be used to receive audio signals, requests, and inputs from the device 100 that are processed as requests, as described below.

[0035] As noted, the device 100 can operate in a low-power mode to conserve energy when not in use. A power circuit 108 can be provided for controlling power levels within the device 100 under the direction of the processor 102. In low-power mode, most of the device 100 systems are shut down and only a few components are operated by the processor 102. One such component is the wake-up word detector 112 described below. The microphone 106 also operates in low-power mode to continuously monitor the environment surrounding the device 100 for audio input.

[0036] In an embodiment, the device 100 can operate at 0.3 watts to 0.5 watts when in low power mode and between 5 watts and 10 watts when in its full power mode, but it should be understood that in other embodiments the device 100 can operate at different power levels in idle or active states. In one example, when the wake word detector detects the presence of any one of the one or more current wake words 120 in the input stream, the processor 102 can exit low power mode and instruct the power circuit 108 to power the device. Upon completion of the user request, the device may return to idle after a predefined time period (e.g., 2-10 seconds), and the processor 102 can instruct the power circuit 108 to switch back to low power mode. In some embodiments, such as where the device 100 is plugged into a power outlet or has a large battery, the power circuit 108 can be omitted and there is no low power mode. However, detection of the wake word is still required to activate the device so that it is ready to listen for queries.

[0037] In the embodiment shown, a wake word detector 112 is present on the device 100. Running the wake word detector locally may be a preferred embodiment. The operation of the word detector is driven by a data structure 120 containing one or more current wake words, one or more wake word detectors, and their associated parameters. Detection of any wake word triggers a transition to the active state.

[0038] A wake-up word detector for a wake-up word can be implemented by a classifier with 2 results (either the wake-up word matches or does not match). In some embodiments, multiple wake-up words are used simultaneously. Parallel detection of any of the N wake-up words can be implemented by a monolithic classifier with N+1 results, one result for each wake-up word and one result for failing to match any wake-up word. In such embodiments, the highest scoring wake-up word may be the one that activates the device. In other embodiments, parallel detection of multiple wake-up words can be implemented by running multiple wake-up word detectors in parallel from the same incoming audio stream. In such embodiments, the earliest matched wake-up word will be the one that activates the device. The possibility of using parallel detectors in this way is a big advantage for using dynamic wake-up words, because the addition of new wake-up word detectors can be implemented without taking into account pre-existing detectors.

[0039] In any of the mentioned detector embodiments, it should be remembered that the private wake word requires positive speaker verification before the device can be activated. The speaker verification engine 114 can run continuously, achieving low latency at the expense of power. If the speaker verification engine 114 is triggered only when the private wake word matches, power consumption can be lower.

[0040] The voice-enabled device 100 may further include a memory 104 that may store algorithms that may be executed by the processor 102. According to an example embodiment, the memory 104 may include random access memory (RAM), read-only memory (ROM), cache, flash memory, a hard disk, and / or any other suitable storage component. Figure 1 As shown in , in one embodiment, the memory 104 may be a separate component in communication with the processor 102 , but in other embodiments, the memory 104 may be integrated into the processor 102 .

[0041] Memory 104 may store various software applications that are executed by processor 102 to control the operation of device 100. Such applications may include, for example, a wake-up word detector 112 for detecting wake-up words in received utterances and a speaker verification engine 114 for verifying the speaker. When the matching wake-up word is private, speaker verification engine 114 is required to verify the speaker's ID. This will be described in more detail later.

[0042] The memory 104 may also store various data records, including, for example, one or more wake-up words 120 and one or more user voice prints 122. Each of these will also be described in more detail.

[0043] The device 100 may further include communication circuitry such as a network interface 124 for connecting to various cloud resources 130 via the Internet. One such resource may be one or more speech recognition and detector building servers 150, also referred to herein as servers 150. Figure 2 An example of the server 150 will be described.

[0044] Figure 2 is a schematic block diagram of an embodiment of server 150. As noted, in other embodiments, server 150 may include multiple servers, collocated or otherwise. Figure 9 A more detailed description of an example server 150 is described, but in general, server 150 may include a processor 152 configured to control the operation of server 150 and facilitate communications between various components within server 150. Processor 152 may include a standardized processor, a dedicated processor, a microprocessor, etc., that executes instructions to control server 150.

[0045] The server 150 may further include a memory 154 that may store algorithms that may be executed by the processor 152. According to an example embodiment, the memory 154 may include RAM, ROM, cache, flash memory, a hard disk, and / or any other suitable storage component. Figure 2 As shown in , in one embodiment, the memory 154 may be a separate component in communication with the processor 152 , but in other embodiments, the memory 154 may be integrated into the processor 152 .

[0046] The memory 154 can store various software applications that are executed by the processor 152 to control the operation of the server 150. Such applications may, for example, include a speech recognition engine 162 for transcribing speech. The application may further include a general parser 155 and a general query fulfillment engine 157 for processing general (non-WAD) queries. The application may further include a wake-up word assignment instruction processor (WAD processor 164) for processing dynamic wake-up word assignment requests. Speech recognition is complex, but many technologies are well-established and do not need to be described in any detail here. It is sufficient to know that in most embodiments, the speech recognition engine 162 has a front end that can produce a phonetic transcription of its input, while the full ASR engine 162 produces a text transcription of the input. When the present disclosure refers to the ASR engine 162, this may refer to the ASR front end or the entire ASR engine, depending on whether phonetic output or text output is required.

[0047] The WAD processor 164 can have software components including a WAD parser 166, a detector builder 168, and a registration engine 170. The components of the WAD processor 164 are described in more detail below. Specifically, the detector builder 168 can build a wake word detector according to one of the methods described below. The registration engine 170 reads and writes the wake word data record 120 representing the wake word used by the device 100 along with its associated detector and parameters.

[0048] The server 150 may further include communication circuitry, such as a network interface 156 for connecting to the cloud resources 130 via the Internet, including to the client devices 100. As indicated, the server 150 may communicate with multiple client devices 100, each of which may be a client device. Figure 1 and configured as described below.

[0049] Now refer to Figure 3 The flowchart of FIG. 1 describes the operation and interaction of the client device 100 and the server 150 to identify a wake word and fulfill a dedicated query to set a new wake word. The diagram is divided between a left column showing modules running locally on the device 100 and a right column showing modules running on the remote server 150 in the illustrated embodiment. In other embodiments, some or all of the components shown in the right column may actually run locally on the device 100.

[0050] At step 200, the device 100 is in an idle state and cannot process voice requests. In this state, it continuously tries to identify a wake word using the wake word detector 112. The device can be in a low power mode to conserve energy while in the idle state. The device 100 remains idle until the wake word detector 112 identifies a wake word in the incoming audio stream. One or more detectors can be used to continuously test one or more wake words against the audio input. The current wake word (and corresponding detector) is looked up in the device's local wake word data structure 120.

[0051] Recall that if the speaker verification test fails, the private wake word is not considered to be matched in the audio input. When the wake word matches, the device exits the idle state and enters the active state. The audio input that begins with the end of the matching wake word and ends with the end of utterance (EOU) is a spoken query. The EOU can be a pause in speech, or a tap, a button press, or a release. The spoken query is given as input to the speech recognition engine 162, which creates a transcription of the spoken query in step 216. In an embodiment, the ASR engine 162 can run locally on the device 100 or remotely on the server 150. In some cases, the speech recognition engine 162 can generate one or more phonemes and / or text transcriptions of the wake word or query, and a score for each transcription indicating the confidence level of each transcription. The ASR algorithm can use any combination of signal processing, hidden Markov models, Viterbi search, phoneme dictionaries, and (possibly recursive) neural networks to generate transcriptions and their confidence scores.

[0052] In general, a virtual assistant can handle a wide variety of queries, including requests for information and commands instructing the client device 100 to perform an action. The manner in which a virtual assistant understands a query varies widely between known embodiments. In the illustrated embodiment, non-WAD queries are identified in step 218 when they are parsed and interpreted by the general query parser 155. Specifically, after the speech recognition engine 162 generates a transcription of the query, the general query parser 155 determines the structure (syntax) and meaning (semantics) of the query. In embodiments, this task can be performed remotely on the server 150 or locally on the device 100. Parsing and processing spoken queries can employ known algorithms for processing queries. Such systems are disclosed, for example, in U.S. Patent No. 10,217,453, entitled “Virtual Assistant Configured by Selection of Wake-Up Phrase,” and U.S. Patent No. 10,347,245, entitled “Natural Language Grammar Enablement by Speech Characterization,” both assigned to SoundHound, Inc., headquartered in Santa Clara, California, which are incorporated herein by reference in their entirety.

[0053] One specific type of query relevant to the present technology is a query requesting that a new wake word be assigned to the device 100. Figure 3For the purpose of this invention, these specific queries, referred to herein as wake word assignment instructions or WADs, are processed (parsed) in step 218 by a dedicated parser - the WAD parser 166. As noted, the speech recognition engine 162 delivers both phonemic transcription and text transcription. The WAD parser 166 can use the text transcription according to a known NLU algorithm to identify the syntax of the instruction. The NLU algorithm can use one or more grammatical patterns to identify the meaning of the instruction portion of the query, and in some cases, also identify the meaning of the wake word portion. However, in general, the wake word can be any word or phrase. In special cases, the wake word may also be a common word or phrase or a known name. In some embodiments, the speech recognition engine 162 can use a language model to increase the transcription score of the wake word that may be used. The NLU algorithm can also play a role in parsing the wake word. However, the wake word can be any speech segment, i.e., a phonemic wildcard, and they are ultimately delimited (segmented) by the instruction grammar of the words surrounding the wake word. It should be understood that various schemes can be used to determine the presence of the wake word and the presence and meaning of the instruction words (phrases) before or after the wake word. It is worth noting that in all such schemes, as the wake word segment is determined, its phonemic transcription becomes available for further processing by the detector builder 168.

[0054] exist Figure 3 In step 220, the query identified by the general query parser 155 is processed for further processing (i.e., fulfillment) by the general query fulfillment engine 157, which acts in the manner desired by the specific virtual assistant. No changes to the "host" virtual assistant are required to implement dynamic wake words. Figure 3 In the exemplary embodiment shown in FIG, after a regular query is processed by the general query fulfillment engine 157 in step 220, the device returns to the idle state. In a variant embodiment not shown, the device remains active (i.e., it accepts queries without a wake word) for a certain period of time (i.e., a few seconds) before returning to the idle state. One way to implement this variant is to return to the active state, but ensure that a timeout period is set when entering the active state, at the end of which the device returns to the idle state.

[0055] When the query is a WAD, the general query parser 155 fails to identify it in step 218. Instead, the WAD parser 166 is able to parse it in step 224 to determine the new wake word and any optional parameters associated with the instruction. In variant embodiments, the WAD parser 166 may be executed before the general query parser 155, or alternatively, both parsers 155 and 166 may be part of a single unified parser. These are Figure 3 A slight variation of the control flow shown in .

[0056] A word about failure: If both parsers 155 and 166 fail to recognize the query, the device will return to the idle state - most likely after issuing an appropriate error message. For example, failure may occur if the speech recognizer 162 cannot reliably determine the transcription of the query. This is not possible when the system allows multiple transcription hypotheses with different scores. Failure may also occur if the query is ungrammatical: when both parsers 155 and 166 cannot recognize the query, the query may fail for syntactic reasons. Other failures may occur if the query is grammatically correct, but its interpretation (meaning) cannot be reliably determined: the query fails for semantic reasons. In addition, a correctly interpreted query may fail during its execution (called fulfillment of the query). Fulfillment of regular queries is performed by the regular query fulfillment engine 157. Fulfillment of WADs consists of the detector builder and the registration engine 170, both of which may exhibit their own failures.

[0057] The WAD parser 166 collects information from the query, including the instruction and its parameters. In addition to adding wake-up words, there are other actions related to wake-up words, such as deleting wake-up words, listing wake-up words, restoring wake-up words to the previous state remembered in the wake-up word record 120, etc. This disclosure focuses on adding wake-up words because it is technically the most difficult part; other related actions are easy to describe. Parameters (e.g., public / private, and exclusive / inclusive) and wake-up words are also determined by the WAD parser 166. The detector builder 168 checks whether the wake-up word is acceptable for use. For example, if the requested wake-up word is too short, or too vague, or too close to a member of a list of unwanted wake-up words (such as offensive words), or is a pre-existing wake-up word for device 100, a new wake-up word detector may not be built. In such cases, the detector builder 168 exits due to failure, and a message can be output to the user via the device 100 to convey the fact that the request cannot be fulfilled.

[0058] If the new wake word is accepted, the detector builder 168 continues to build a detector for the wake word in step 226. Figure 5-8 Further details of various embodiments for building a new wake word detector by the detector builder 168 are described. Upon success of the detector builder, the wake word, its wake word detector, and associated parameters are handed over to the registration engine 170. When the detector builder is run on the server 150, this data (including the new detector) is downloaded to the device 100. The registered detectors, wake words, and parameters are stored on the device 100 in a data structure 120 containing one or more wake words and associated data.

[0059] As Figure 3Flowchart example of the operation of the client device 100 may receive an utterance:

[0060] "OK AGENT, RESPOND TO OK JARVIS"

[0061] where "OK AGENT" is the current wake-up word. The rest of the utterance following the wake-up word "OK AGENT" is the query "RESPOND TO OK JARVIS". The query can be uploaded to the server 150 and transcribed using the speech recognition engine 162 in step 216. The WAD parser 166 recognizes the query as an instruction and extracts the wake-up word in step 224. In this example, the query "Respond to OK Jarvis" is a wake-up word assignment instruction requesting that the wake-up word speech audio segment "OK Jarvis" be assigned as a new wake-up word. After successfully parsing this wake-up word assignment instruction into its wake-up word and parameters in step 224, the wake-up word detector builder 168 must generate a detector for the new wake-up word in step 226, either by building a completely new wake-up word detector or by locating a pre-existing detector for the wake-up word. As will be described later, this operation can be implemented in many ways. The new wake word detector, new wake word, and associated parameters are then registered in step 228 by the registration engine 170, which modifies the wake word 120 data structure that holds the current wake word, detector, and associated parameters.

[0062] Figure 3 One embodiment of processes performed on either the client device 100 or the server 150 is shown. In the illustrated embodiment, all processes in the left column (labeled "Client Device") are performed on the device 100, and all processes in the right column (labeled "Server") are performed on the server 150. In alternative embodiments, some or all of the processes shown in the right column may actually be performed on the client side instead. For example, steps 216 and 224 (recognizing WADs locally) may be performed locally on the device 100, while general query parsing in step 218 and general query processing in step 220 may be performed on the server 150. In such cases, the WAD parser 166 runs before the general query parser 155. In these cases, the detector builder is configured to run partially or entirely on the device 100. In a first embodiment, the detector builder has access to the computing power and labeled audio database of the server 100 when building the detector. This is exemplified by the segmented approach to detector construction described later. In a second embodiment, the detector builder is completely native; this will be exemplified by the transcription method for detector construction described later.

[0063] As noted, wake-up word assignment involves instructions, a new wake-up word speech audio segment, and optional parameters that control the processing of the new wake-up word. In one example, the parameters may be related to whether the newly assigned wake-up word is public or private. A public wake-up word can be used by anyone to wake up the device 100 and gain access to the resources of the device 100. On the other hand, a private wake-up word is personal to the user who created it, so that the device 100 will only be activated when the wake-up word is spoken by the wake-up word creator, and will remain in an idle state when spoken by others. In another example, the parameters may be related to whether the new wake-up word will be added to the set of existing wake-up words or will replace some or all of the existing wake-up words.

[0064] Now refer to Figure 4 Referring to the flowchart of FIG. 2 , in step 260 , the WAD processor 164 may check whether the wake word assignment instruction includes a parameter stating whether the new wake word is public or private. The following are some examples of parameters that may indicate to the WAD processor 164 that the wake word will be a public wake word. In the following examples of the wake word assignment instruction, the wake word itself has been omitted, leaving only the query portion of the utterance.

[0065] "RESPOND TO OK VICTORIA" (assuming default value of "Public")

[0066] “RESPOND TO PUBLIC WAKEWORD OK MADELINE”

[0067] “RESPOND TO OK MADELINE PUBLIC WAKEWORD”

[0068] “WE WILL CALL YOU HEY JACKSON”

[0069] In the first example, no parameters are provided, causing the WAD processor 164 to assign the new wake-up word "OK Victoria" as a public wake-up word by default. In the second example, the parameter wording "publicwakeword" is explicitly stated before or after the speech audio segment and the corresponding attributes are set. The WAD parser 166 can look for such predefined parameter wordings so as not to treat them as part of the speech audio segment. In the fourth example, the "public" parameter setting is not explicit, but can be inferred from the use of the plural subject "we" (as opposed to the singular "I") in the wake-up word assignment instruction, which cleverly designates the new wake-up word "Hey Jackson" as a public wake-up word. A variety of other examples of wake-up word assignment syntax can be imagined, in which the explicit or implicit wording of parameters in the spoken wake-up word assignment instruction indicates that the new wake-up word will be public.

[0070] Alternatively, the wake word assignment instruction may include a parameter indicating that the wake word will be a private wake word. Following are a few examples.

[0071] "RESPOND TO OK VICTORIA" (assuming default value of "private")

[0072] “ PRIVATELY RESPOND TO OK JOSEPHINE( in private Respond to good Josephine)"

[0073] “RESPOND TO OK JOSEPHINE PRIVATELY ( in private Respond to good Josephine)"

[0074] “RESPOND TO PRIVATE WAKEWORD HEY CHRISTOPHER Private wake-up word Hey Christopher responded)"

[0075] “ I WILL CALL YOU TABATHA (I will call you Tabatha)

[0076] “ YOUR NICKNAME IS PENELOPE( Your nickname It's Penelope)

[0077] The first example shows that the default value can alternatively be to make the wake word private when no parameters are provided. In the next four examples, the parameter wording "privately" or "privatewakeword" or "nickname" is used as part of the query before or after the voice audio segment to convey the private parameter setting. In the last example, the private setting can be inferred from the use of the first-person singular subject pronoun "I" at the beginning of the query. A wide variety of other examples can be created in which explicit wording or contextual parameters within the spoken wake word assignment instruction indicate that the new wake word is intended to be private.

[0078] If a public / private parameter is detected in step 260, the WAD processor 164 may check in step 262 whether a private parameter exists. If not, the WAD processor 164 may treat the new wake word as public when stored. On the other hand, if the parameter indicates in step 262 that the wake word is private, the WAD processor 164 may execute step 264 to create a voiceprint of the speaker's voice and possibly additional speaker verification data. The user voiceprint information may be stored on the device 100 in the voiceprint data structure 122, or may be obtained by the device 100 from cloud user records for later speaker verification.

[0079] In step 268, the WAD processor 164 may associate speaker verification data with the new wake word and the detector by calculating a voiceprint. The WAD processor 164 may treat the new wake word as private and store the speaker verification data. This verification data (such as a voiceprint) may be stored in the user voiceprint 122 in the memory 104 of the individual user.

[0080] In operation, after receiving an utterance and confirming the presence of a private wake-up word, the processor 102 may further use data from the user voiceprint 122 associated with the private wake-up word to check whether the speaker is the same as the speaker who created the private wake-up word. If there is a match, the processor 102 may signal the power circuit 108 to power on the device 100. If there is no match, the processor may ignore the wake-up word and remain in an idle state.

[0081] Instead of, or in addition to, the public / private parameter, the wake-up word assignment instruction may also include a parameter regarding whether the new wake-up word is to be added to a set of one or more existing wake-up words or to replace one or more existing wake-up words in the set of wake-up words. In step 272, the WAD processor 164 checks whether the wake-up word assignment instruction includes a parameter regarding whether the new wake-up word is to be added to an existing wake-up word or to replace an existing wake-up word. The following are several examples of parameters that may indicate to the WAD processor 164 that a wake-up word is to be added to an existing wake-up word.

[0082] “RESPOND TO OK FREDERICK”

[0083] "RESPOND TO ADDED WAKEWORD ESTEBAN"

[0084] “ALSO RESPOND TO OK SAMUEL”

[0085] In a first example, no parameters are provided, causing the WAD processor 164 to default to adding the new wake word "Frederick" to the other existing wake words 120. Thus, the user can wake up the device 100 with the wake word "OK Frederick" in addition to one or more previously existing wake words 120 in memory 104. Here, the new wake word is said to be additive to the existing wake words. In a second example, the parameter "added wakeword" can be a predefined phrase that is explicitly stated before or after the speech audio segment to make the new wake word additive to the existing wake words. When parsing the received wake word assignment instruction, the WAD processor 164 can look for such predefined parameters so as not to treat them as part of the speech audio segment. In a third example, the parameter "also" (part of "also respond to" or "respond also to") explicitly requests that the new wake word be added to the existing wake words. A variety of other examples are contemplated in which the expression and / or context parameters within the spoken wake word assignment instruction indicate that the new wake word is additive to the wake word set of one or more existing wake words.

[0086] Alternatively, the wake-up word assignment instruction may include a parameter indicating that the wake-up word will replace one or more of the current wake-up words. Here are a few examples (parameters are underlined to emphasize):

[0087] "RESPOND TO OK BRITTANY"

[0088] “RESPOND TO OK BRITTANY EXCLUSIVELY ( Exclusively Respond to good Brittany)"

[0089] “RESPOND ONLY TO MARLENA( only Respond to Marina)

[0090] The first example shows that the default value can alternatively be to make the wake word exclusive and remove the previous wake word when no parameter is provided. In the second example, the parameter "exclusively" can be a predefined phrase that is explicitly stated following the voice audio segment. In the third example, the parameter "only to" can be a predefined phrase that is explicitly stated before the voice audio segment. Instead of a predefined phrase indicating the exclusivity of the new wake word in the second and third examples, the exclusivity of the new wake word can be inferred based on the context of the parameters in the wake word assignment instruction. A variety of other examples are envisioned in which expressions and / or context parameters within the spoken wake word assignment instruction indicate that the new wake word will be private.

[0091] If the exclusive or additive parameter is detected in step 272, the WAD processor 164 can check in step 274 whether the parameter indicates that the new wake word is additive. If so, when the new wake word is stored, a flag can be set in step 276 to store the new wake word in addition to the existing wake word. On the other hand, if the parameter indicates in step 274 that the wake word is replacing one or more existing wake words, the WAD processor 164 can check in step 280 whether multiple wake words are stored. If so, the processor 102 can generate a query as to which of the multiple wake words will be replaced. Alternatively, steps 280 and 282 can be skipped, and all existing wake words can be replaced by default.

[0092] In the case where one or more wake-up words are being replaced, a flag may be set in step 286 that indicates which of the stored wake-up words is being replaced when the new wake-up word is stored. It may happen that the user does not have permission to replace one or more of the wake-up words, which may be determined, for example, based on data in the user's voiceprint 122 stored in the memory 104. In this case, the processor may replace only those wake-up words that the user has permission to replace.

[0093] It will be further appreciated that parameters relating to both private / personal and additive / exclusive can be provided in a single wake word assignment instruction. The following are a few examples.

[0094] “I WILL ONLY CALL YOU OK ROBERT”

[0095] “WE WILL ALSOCALL YOU OK NATALIA (We will also call you Natalia)”

[0096] In the first example, the new wake-up word "OK Robert" is designated as a private wake-up word using the first-person singular subject pronoun "I" at the beginning of the wake-up word assignment instruction, and the use of the word "only" makes it an exclusive wake-up word. In the second example, the new wake-up word "OK Natalia" is designated as a public wake-up word using the first-person plural subject pronoun "we" at the beginning of the wake-up word assignment instruction, and the use of the phrase "also" makes it an additive wake-up word. Although the wake-up words in the above examples are common names for people, it should be understood that the wake-up words 120 may include any word or phrase, meaningless or otherwise. In further embodiments, it is conceivable that the new wake-up word is formed by sounds other than speech, such as, for example, a doorbell, an alarm, or a drum beat, among several possibilities.

[0097] While the above discussion of parameters relates to two specific aspects (private / personal and additive / exclusive), it should be understood that parameters related to other aspects of generating new wake-up words and detectors may be provided in addition to or in lieu of the above examples.

[0098] In addition, in the above example, the parameters are provided in a single utterance including the wake-up word assignment instruction. In another embodiment, the parameters can be established in a modal discourse between the user and the device 100. Specifically, the user can initially issue a wake-up word assignment instruction without parameters. Thereafter, the WAD processor 164 can prompt the user to provide additional parameters and information in the following manner: the processor 102 generates text, the text is converted to speech by a TTS algorithm and played through the speaker 136. The following provides an example of how to prompt parameters for such a modal dialogue and how to provide parameters in such a modal dialogue.

[0099] (U)RESPOND TO JARVIS

[0100] (D) Got it. Can you say that three times?

[0101] (U)JARVIS…JARVIS…JARVIS(Jarvis…Jarvis…Jarvis)

[0102] (D) OK, I NOW RESPOND TO JARVIS (OK, I now respond to Jarvis)

[0103] (D) SHOULD JARVIS BE PRIVATE OR PUBLIC? (Should Jarvis be private or public?)

[0104] (U)PUBLIC (Common)

[0105] (D)SHOULD I SAVE PREVIOUS WAKEWORDS?

[0106] (U)NO

[0107] (D)ARE YOU SURE YOU WANT TO FORGET THE DEFAULT PUBLIC WAKEWORD OKAGENT?

[0108] (U)NO,TELL ME THE WAKEWORDS

[0109] (D)SURE.JARVIS IS A PUBLIC WAKEWORD AND OK AGENT IS APUBLIC WAKEWORD.

[0110] (U)DISABLE THE PREVIOUS WAKEWORDS (Disable the previous wake words) or

[0111] (U)DISABLE OK AGENT (Disable OK Agent)

[0112] (D)OK AGENT IS NOW DISABLED (OK Agent is now disabled)

[0113] The above is an example of a conversation between the user (U) and the device 100 (D). As can be seen, the device 100 can receive wake word assignment instructions and prompt the user to repeat the wake words several times. Then, the device 100 can prompt the user to specify whether the new wake word is public or private, and whether it is additive or exclusive.

[0114] For a device 100 controlled by a single owner, the device can set its (private or public) wake word by the device owner, and other users have no control over it. However, in the case where the device 100 is a shared device controlled by multiple users, it may be desirable to save the public wake words after replacing them. Specifically, when instructing to replace all previous wake words on the device, there is a risk that users (possibly even the user who issued the instruction) will be locked out and unable to activate or access the resources of the device 100.

[0115] Therefore, a recovery procedure can be provided to avoid the adverse consequences from such a deadlock. In this case, the device 100 can be restored to a previous state in which the previous wake word access rights are restored. This provides the lowest level of default functionality for users who have not issued an instruction for a new wake word and may not even be aware that the wake word they are using has been replaced by a new wake word. In some systems, this can be achieved by a hard reset - either resetting to factory settings or resetting to the working configuration of a previously saved set of wake words.

[0116] In an embodiment, by using a single instruction and / or a modal dialogue, a complex mix of wake word replacement strategies, wake word addition strategies, and / or procedures for saving and restoring wake word set configurations can be supported together. For devices that have a GUI in addition to audio, such as a dedicated reset button, these other interfaces can be used to replace the audio signal or as a supplement to the audio signal for wake word recovery. For devices that do not have a GUI in addition to audio, a full reset of the device state can be achieved via the audio interface (microphone and speaker) and / or by periodic power cycling.

[0117] The following is another example of a modal dialogue for setting a dynamic wake word (including restoring a replaced wake word).

[0118] (U) RESPOND TO PUBLIC NAME BUTLER3 (Respond to the public name BUTLER3)

[0119] (D) OK, BUTLER THREE, IS THAT RIGHT? (Okay, BUTLER three, is that right?)

[0120] (U) YES

[0121] (D) SHALL I KEEP MY PRIVATE NAME, JARVIS? (Should I keep my private name, JARVIS?)

[0122] (U) NO (No)

[0123] (D) AND KEEP MY PUBLIC WAKEWORD OK AGENT? (And keep my public wake word okay agent?)

[0124] (U) PLEASE DISABLE IT (Please disable it)

[0125] ... Time passes...

[0126] (U)BUTLER3, WHAT ARE THE WAKEWORDS?

[0127] (D) BUTLER THREE IS AN ACTIVE PUBLIC WAKEWORD, AND OK AGENT IS ADISABLED PUBLIC ONE.

[0128] ...time passes...

[0129] (U)BUTLER3, RESTORE PREVIOUS WAKEWORDS

[0130] (D) OK AGENT IS NOW A PUBLIC WAKEWORD (OK agent is now a public wake word).

[0131] Likewise, other examples of modal dialogue are envisioned.

[0132] The above describes a process for setting a new wake-up word using audio instructions via one or more microphones 106. In alternative embodiments, the device 100 may have other interfaces, such as a GUI or a physical (touch) interface that is configured to implement the wake-up word assignment instructions in whole or in part. For example, the device may have a hold button that can be pressed to enter a new wake-up word and released at the end of the voice audio segment. In addition, releasing the button immediately after pressing it may cause the device to speak its current wake-up word and corresponding state (e.g., public or private).

[0133] As described above, step 226 ( Figure 3 ) involves building a wake-word detector for dynamic wake-words. As noted in the "Background" section, an effective wake-word detector can be designed for factory-installed wake-words by collecting a large dataset of predefined wake-words and then using that dataset to train an acoustic model, such as a neural network. This technique is not feasible for dynamic wake-words because such a dataset typically does not exist for dynamically selected wake-words.

[0134] The present technology overcomes this problem by using one or more methods of immediately building a wake word detector for any valid dynamic wake word. As noted above, the term "immediately" is used here to mean that a new wake word detector can be built within a few seconds (e.g., 2 to 5 seconds) after the user has spoken the wake word assignment instruction. In another example, the term "immediately" as applied to the time it takes to build a detector can mean the time it takes to complete a confirmation response to a WAD. For example, a detector would be considered to be built "immediately" if it was built before providing the following response: "I will now respond to Jarvis" or "Josephine is now a public wakeword", etc.

[0135] In an embodiment, the wake-up word detector for a new wake-up word is provided by the WAD processor 164 ( Figure 2 ) is constructed by the detector builder 168. As pointed out above, although Figure 2 The Detector Builder 168 is shown on the server 150 , but components of the Detector Builder 168 may reside and be implemented on the server 150 , the device 100 , or a combination of the server 150 and the device 100 .

[0136] Wake-up word sampling method for detector construction

[0137] In one embodiment, the detector builder 168 can use a method referred to herein as a wake-up word sampling method to build a wake-up word detector. In this method, multiple sample utterances of a new wake-up word are collected from the user and then used to build a new wake-up word detector by training a classifier locally. Figure 5 The flowchart describes the method.

[0138] In steps 290 and 292, the detector builder 168 uses a modal dialog to ask the user to provide additional audio samples of the new wake word. Some embodiments may request one wake word at a time; other embodiments may leave it open so the user can provide multiple samples. Receipt of a sufficient number of samples of the new wake word (e.g., four or more) may be confirmed in step 292. In other embodiments, the number may be less than four. The initial audio samples of the wake word, along with the additional audio samples collected through steps 290 and 292, are used in step 296 to build a classifier (such as a neural network (NN) classifier) that will serve as a wake word detector. The collected samples serve as positive examples of the wake word. To avoid false positives, it may be useful to add negative examples during training. When used, negative examples can be generated in a number of ways. In one embodiment, the audio samples of the wake word can be transcribed into phoneme sequences by the ASR front end; these sequences can then be slightly perturbed to cause near misses. Speech synthesis can be used to obtain negative audio samples from near miss phoneme sequences. In a simpler variation of the previous embodiment, a single phoneme sequence is used, namely the phoneme sequence from the speech recognition step 162. In this variation, no additional transcription step is required. This is particularly convenient when step 162 is performed on a server, as the device 100 does not have to support the ASR function 162.

[0139] In summary, the wake-up word sampling method is divided into three steps:

[0140] 1. Collect positive audio samples of the wake-up word;

[0141] 2. Optionally generate some negative audio samples of the wake-up word; and

[0142] 3. Train a classifier that will act as the desired wake-up word detector.

[0143] These steps can be performed locally on the device 100, or they can be performed on a server. Because the training sample set is very small, it is computationally feasible to implement the wake-up word sampling method locally on a device with limited resources. The small training data set provides limited reliability. Because the positive sample comes from a single speaker, the wake-up word sampling method can prove to be most reliable when used with a private wake-up word that is used only by the specific user who created the private wake-up word. Therefore, the device 100 can use the speaker verification engine 114 to verify that the current speaker's voice matches the voice of the speaker who created the private wake-up word. The device 100 stores the user voiceprint 122 to support this functionality in conjunction with the speaker verification engine 114.

[0144] Continuous transcription method for detector construction

[0145] In another embodiment, the detector builder 168 can implement the detector using a method referred to herein as a continuous transcription approach. In this approach, the detector algorithm relies on the embedded speech recognition engine 162 (or more precisely, the ASR front end) to generate a continuous phoneme transcription of the input audio. To achieve the low latency required for the detector, the speech recognition front end can run locally on the device 100. Figure 6 The flowchart of describes such an embodiment. In step 310, the speech recognition front end maps the incoming audio stream to a phoneme stream, which is a continuous phoneme transcription of the input. In step 312, the wake-up word detector attempts to match consecutive segments of the incoming phoneme stream with the phoneme sequence of the active wake-up word. This operation is performed continuously for each possible alignment of the phoneme sequence of the wake-up word relative to the phoneme stream. When an alignment hypothesis is started, the hypothesis remains valid as long as the phoneme match is maintained. If there are several active wake-up words in the wake-up word 120 data structure stored in memory 104, the incoming phoneme stream is compared in this way with each stored wake-up word in parallel. The above steps can be applied incrementally each time a new phoneme appears in the incoming phoneme stream. The first match detected in step 316 (the alignment hypothesis completed in the match) will trigger the wake-up word detector in step 318. If the phoneme match fails at any time during the alignment hypothesis, the process returns to step 312 to look for a new phoneme sequence.

[0146] In an embodiment, multiple phoneme transcriptions of an incoming audio stream may be considered in parallel, for example, using a phoneme lattice data structure. Similarly, in an embodiment, multiple phonetic transcriptions of a wake word may be considered in parallel, for example, using a phoneme lattice data structure. Whenever multiple hypotheses are considered, they may have associated probabilities or scores. In an embodiment, a wake word phoneme sequence is associated with a time component of each phoneme, and a phoneme alignment receives a score associated with the amount of time stretch between the wake word phoneme and the incoming phoneme. In an embodiment, a low alignment score, or a low probability or score for an alternative hypothesis may result in the alignment hypothesis being discarded.

[0147] This approach to providing a wake word detector is advantageous because it does not require training based on a stored dataset—whether the dataset is a large remote dataset on a server or a small sample set of dynamic wake word data collected when needed. This approach is well suited for use with common wake words by different people and for robustness to noise because the speech recognition engine 162 or its ASR front end that generates the phoneme stream from the incoming audio stream is pre-trained for a wide variety of speakers and situations. Depending on the battery technology, this approach may be best suited for a device 100 that is plugged into an electrical outlet or can draw the battery power required to perform continuous speech transcription, but this need not be the case in other embodiments. Matching the continuous phoneme transcription input to the stored wake word phoneme sequence can be performed quite efficiently.

[0148] The main variation of the method just described is to use continuous text transcription instead of continuous phoneme transcription. The full speech recognition module 162 may usually involve a large phoneme dictionary and a large language model, both of which consume a lot of memory. In the present case, it is possible to completely ignore the language model (thus greatly reducing space) and use a reduced phoneme dictionary - the default phoneme-to-text converter without considering exceptions.

[0149] The transcription method itself does not involve training, such as training a NN, but it does "build" a detector, consisting of a phoneme sequence matching algorithm and one or more target phoneme sequences.

[0150] Wake-up word segmentation method for detector construction

[0151] In yet another embodiment, the detector builder 168 may construct a wake word detector using a method referred to herein as a wake word segmentation method. This method starts with a phoneme transcription of the new wake word, which may be obtained from the speech recognition step 216 prior to step 226. Figure 7 Flowchart and Figure 8 A diagram is used to describe this method.

[0152] After the received utterance is parsed by the WAD parser 166, the phonemic wake word transcription of the parsed utterance can be tested to see if there is a wake word detector that has been trained and cached for the entire wake word in step 320. Such a detector can be cached on the server 150, but may alternatively be cached on the device 100 or a third-party server.

[0153] For example, Figure 8The phoneme transcription of the wake word from the WAD is shown. In this example, the wake word "HEY CHRISTOPHER" can be spelled phonetically as "HH EY1 KR IH1 ST AH0 F ER0" using the CMU phoneme alphabet. CMUP is the standard phoneme alphabet for English. Other phoneme alphabets, such as IPO, the International Phonetic Alphabet, can be used to define phoneme sequences for wake word detectors in English or other languages. Step 320 tests whether there is a cached detector that has been trained for this entire phoneme sequence. If so, the cached detector is downloaded in step 322 and used as the new wake word detector for the new wake word "HEY CHRISTOPHER".

[0154] When no cached detectors exist for the entire new wake-up word, the wake-up word may be segmented into a plurality of phoneme segments in step 326. The segmentation of the wake-up word portion into a sequence of phoneme segments may be implemented in any of a variety of ways, including segmenting the wake-up word portion into word or syllable groups, individual syllables, or finer divisions. As a simple example, Figure 8 Shown are the root segment, ie, segment 1 where the entire wake word is a single segment, and the second segment, ie, segment 2 where the wake word is broken down into phoneme segments from the individual words "HEY" and "CHRISTOPHER".

[0155] In step 328, the detector builder 168 checks whether a detector already exists and is cached for each phoneme segment in the current segment. If so, these detectors for each phoneme segment are assembled together in the order of consecutive phoneme segments in step 348. These consecutive detectors are then downloaded and used as new wake word detectors.

[0156] If at any time the detector builder 168 determines that a phoneme segment in a given segment does not have a corresponding cached detector, the engine 168 next checks in step 330 whether there are further possible partitions of the wake word into phoneme segments. For example, Figure 8 Another segmentation is shown, namely segment 3 where the wake word is further divided into syllables. There are syllabification algorithms that automatically segment valid phoneme sequences into syllables. As discussed later, further segmentation is possible. If another such segmentation is possible in step 330, a new segmentation step is taken in step 334, resulting in a new segmentation, which is tested again in step 328 to see if there is a detector cached for each segment in the new instance.

[0157] Whenever the division (segmentation) of the wake word phoneme sequence is completed, the wake word segmentation method builds a new detector for any phoneme segment that does not have a detector cached in memory. To build a new detector for a phoneme segment, this technology relies on access to a database of labeled audio in memory 154 on server 150. Specifically, a collection of audio samples corresponding to a particular phoneme sequence can be retrieved from a database of audio segments labeled by their phoneme transcriptions. In some embodiments, this search can be optimized by using a pre-computed index, such as a tree structure in which nodes are associated with corresponding positions in a corpus of audio segments.

[0158] The detector can then be trained based on positive examples using the retrieved matching segments for the wake word. Negative examples useful for training a yes / no classifier can be obtained in many ways. "Close matches" can be used to avoid false positives; false positives can be obtained by perturbing the segmented phoneme sequences (e.g., by using closely related phoneme sequences). Random audio samples can also be used, which will help improve the classifier's output probability.

[0159] Because it is most efficient to build as few detectors as possible, the detector builder 168 can select the instance of the detector that already has the most cached detectors for its phone segment in step 336. Then, in step 338, the detector builder 168 retrieves a subset of data from the database in memory 154, as described above, to use when training a detector for the phone segment that does not have a cached detector. The subset of data is used to train a detector for this phone segment in step 340. Once the detector is trained for this phone segment, it can be added to the cache in step 344.

[0160] In step 346, the detector builder 168 can check whether there are additional phoneme segments in the selected instance that do not already have cached detectors. If so, a new phoneme segment is selected and steps 338, 340, and 344 are repeated on the new phoneme segment. This process continues until all phoneme segments have cached detectors in step 346. At that time, all cached detectors for each phoneme segment are assembled together in the order of consecutive phoneme segments in step 348. These consecutive detectors are then downloaded and used as new wake word detectors.

[0161] In an embodiment, due to computational reasons or the amount of memory required for the labeled audio database and / or segment detector cache, the training of detectors for phoneme segments in this method (steps 338 and 340) may be performed on server 150. Step 348 of assembling detectors for different phoneme segments in the wake word detector may be done on server 150 and downloaded to device 100, or performed on device 100 itself.

[0162] In an embodiment, the continuous detector assembled in step 348 for each continuous phoneme segment can be viewed as a yes / no classifier that, given an input stream, determines the probabilities of yes (matching its phoneme segment) and no (failed to match). The successful path of the detector is an all-yes path: if each classifier step has a probability above a threshold, and the total probability of the path (the product of the probabilities throughout the path, or the sum of the log probabilities) is above the threshold, then the detector succeeds. The detector is applied to the audio in a number of continuous alignments, such as at every frame at a given frame rate.

[0163] In the above example, the method tests the presence of a detector in the cache for the phoneme segment of the new wake-up word at various levels. Figure 7 The algorithm performs progressive deepening, where the phoneme segments from the wake word are continuously divided. This allows the method to take advantage of a large pre-existing wake word segment detector. But there are simpler algorithms where a specific level of phoneme segmentation is assumed. In one variant, the wake word can be divided into words - assuming that word segmentation of the wake word is available in addition to the phoneme sequence. In another variant, the wake word phoneme sequence can be segmented into syllables. Syllable-level detectors can thus be built and cached as needed. But one might imagine that detectors could be precomputed for every possible syllable. In an embodiment, wake word detectors can be predefined and stored on server 150 for all required defined enumerations of phoneme segments (e.g., for all syllables). This is feasible in principle and should perform well, allowing the task to be completed without having to train any new segment detectors in step 340. But there are a very large number of possible syllables in English or in other languages.

[0164] A syllable is formed of three clusters: an onset, a nucleus, and a coda, where the nucleus cluster consists of vowels and the onset and coda clusters consist of consonants. For example, using the CMU phonemic alphabet, the syllable "STR EE TS" ("streets streets") has a 3-consonant cluster "STR" before a single vowel cluster "EE" and a 2-consonant coda "TS."

[0165] Another variation is to perform subsyllable segmentation as another type of phoneme segment classification for which all wake-up word detectors can be trained. For example, each vowel can be segmented into an initial part that includes an initial consonant cluster and at least the initial vowel, and a final part that includes the final vowel and the final consonant cluster. When a vowel cluster has a length of 1, the vowel is both initial and final. When a vowel cluster has a length of 2, it consists of an initial vowel and a final vowel. When a vowel cluster has a length of 3, some additional rules can be used to decide how to segment the consonant cluster.

[0166] In any of the above embodiments, once a wake word detector has been created, the registration engine 170 can register the detector, including storing or caching the wake word and the wake word detector in memory, even after the wake word for the detector has changed. In that way, if the old wake word is used again, the cached wake word detector can be immediately pulled up.

[0167] In the above embodiments, a single detector can be used to detect the wake-up word even when the wake-up word is a phrase containing multiple words. In other embodiments, a "multi-detector" can be used to detect the wake-up word containing multiple words. When there are N words involved in the wake-up word, the activation module can rely on running N detectors in parallel for multiple words in the wake-up word (each of the N detectors has a binary output - match or failure), or a joint detector can be trained (such as a classifier with N+1 results, one for each possible match and one for failure), or a combination of the two.

[0168] Figure 9 An exemplary computing system 900 is illustrated that may serve as a device 100 or server for implementing embodiments of the present technology. Figure 9 The computing system 900 includes one or more processors 910 and a main memory 920. The main memory 920 stores, in part, instructions and data executed by the processor unit 910. When the computing system 900 is in operation, the main memory 920 may store executable code. Figure 9 The computing system 900 may further include a mass storage device 930 , a portable storage media drive 940 , an output device 950 , a user input device 960 , a display system 970 , and other peripheral devices 980 .

[0169] Figure 9 The components shown in FIG are depicted as being connected via a single bus 990. The components may be connected by one or more data transfer means. The processor unit 910 and main memory 920 may be connected via a local microprocessor bus, while the mass storage device 930, peripheral devices 980, portable storage media drive 940, and display system 970 may be connected via one or more input / output (I / O) buses.

[0170] Mass storage device 930, which may be implemented as a magnetic disk drive or optical disk drive, is a nonvolatile storage device for storing data and instructions used by processor unit 910. Mass storage device 930 may store system software implementing embodiments of the invention for the purpose of loading that software into main memory 920.

[0171] The portable storage medium drive 940 operates in conjunction with a portable non-volatile storage medium such as a floppy disk, a compact disk, or a digital video disk to transfer data to the portable storage medium. Figure 9 The computing system 900 inputs data and code and receives Figure 9 System software for implementing embodiments of the present invention may be stored on such portable media and input to the computing system 900 via the portable storage media drive 940 .

[0172] The input device 960 provides a portion of the user interface. The input device 960 may include an alphanumeric keypad (such as a keyboard) for inputting alphanumeric and other information, or a pointing device (such as a mouse, trackball, stylus, or cursor direction keys). Additionally, as Figure 9 The system 900 shown in FIG. 1 includes an output device 950. Suitable output devices include speakers, printers, network interfaces, and monitors. In the case where the computing system 900 is part of a mechanical client device, the output device 950 may further include servo controls for motors within the mechanical device.

[0173] The display system 970 may include a liquid crystal display (LCD) or other suitable display device. The display system 970 receives text and graphics information and processes the information for output to a display device.

[0174] Peripheral devices 980 may include any type of computer support device to add additional functionality to a computing system. Peripheral devices 980 may include a modem or a router.

[0175] Figure 9 The components included in computing system 900 are those typically found in computing systems that may be suitable for use with embodiments of the present invention and are intended to represent a broad class of such computer components that are well known in the art. Figure 9 The computing system 900 can be a personal computer, a handheld computing device, a phone, a mobile computing device, a workstation, a server, a minicomputer, a mainframe computer, or any other computing device. The computer can also include different bus configurations, networked platforms, multi-processor platforms, etc. Various operating systems can be used, including UNIX, Linux, Windows, Macintosh OS, Palm OS, and other suitable operating systems.

[0176] Some of the above functions may be composed of instructions stored on a storage medium (e.g., a computer-readable medium). The instructions may be retrieved and executed by a processor. Some examples of storage media are memory devices, magnetic tapes, disks, etc. The instructions, when executed by the processor, are operable to direct the processor to operate in accordance with the present invention. Those skilled in the art are familiar with instructions, processors, and storage media.

[0177] It is noteworthy that any hardware platform suitable for performing the processing described herein is suitable for use with the present invention. As used herein, the term "computer-readable storage medium" refers to any one or more media that participate in providing instructions to the CPU for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as fixed disks. Volatile media include dynamic memory, such as system RAM. Transmission media include coaxial cables, copper wires, and optical fibers, including wires of an embodiment comprising a bus. Transmission media can also take the form of sound waves or light waves, such as sound waves or light waves generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, any other physical media with markings or hole patterns, RAM, PROM, EPROM, EEPROM, FLASHEPROM, any other memory chip or cartridge, carrier waves, or any other medium from which a computer can read.

[0178] Various forms of computer-readable media may be involved in delivering one or more sequences of one or more instructions to the CPU for execution. A bus carries the data to system RAM, from which the CPU retrieves and executes the instructions. The instructions received by the system RAM may optionally be stored on a fixed disk before or after execution by the CPU.

[0179] In summary, the present technology relates to a method for modifying a set of one or more wake-up words of a voice-enabled device, comprising: receiving a verbal request from a user; parsing the request into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; and defining a new wake-up word detector to identify the new wake-up word as an activation trigger.

[0180] In another example, the present technology relates to a method for modifying a set of one or more wake-up words for a voice-enabled device, comprising: receiving a verbal request from a user; parsing the request into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; defining a new wake-up word detector to identify the new wake-up word as an activation trigger by dividing the speech audio segment into continuous phoneme segments, comparing the parsed audio segment with a segment dataset used to train an existing ASR algorithm to find a match between the parsed audio segment and a segment in the segment dataset, and training the detector using one or more matching phoneme segments from the existing ASR algorithm.

[0181] In another example, the present technology relates to a method for modifying a set of one or more wake-up words for a voice-enabled device, comprising: receiving a verbal request from a user; parsing the request into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; defining a new wake-up word detector to identify the new wake-up word as an activation trigger by the following steps: obtaining an additional speech audio sample of the speech audio segment, converting the speech audio segment and the additional speech audio sample into a phoneme sequence, and defining a wake-up word detector based on one or more phoneme sequences in the phoneme sequence.

[0182] The above description is illustrative and not restrictive. After reviewing this disclosure, many variations of the present invention will become apparent to those skilled in the art. Therefore, the scope of the present invention should not be determined with reference to the above description, but instead should be determined with reference to the full scope of the appended claims and their equivalents. Although the present invention has been described together with a series of embodiments, these descriptions are not intended to limit the scope of the present invention to the specific form set forth herein. It should be further understood that the method of the present invention is not necessarily limited to the discrete steps or the order of steps described. On the contrary, the present description is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the present invention as defined by the appended claims and that are otherwise understood by those of ordinary skill in the art.

[0183] Those skilled in the art will recognize that an Internet service can be configured to provide Internet access to one or more computing devices coupled to the Internet service, and that the computing devices can include one or more processors, buses, memory devices, display devices, input / output devices, etc. Furthermore, those skilled in the art can appreciate that the Internet service can be coupled to one or more databases, repositories, servers, etc. that can be utilized to implement any of the embodiments of the present invention as described herein.

Claims

1. A method for modifying a set of one or more wake-up words for a voice-enabled device, comprising: receiving a verbal request from a user; Parsing the spoken request into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; as well as Build a new wake-up word detector to identify the new wake-up word as an activation trigger, Among them, building a new wake-up word detector includes: Dividing the new wake-up word into multiple phoneme segments; and For each phoneme segment in the plurality of phoneme segments: Check whether there is a cached detector for the phoneme segment; When there is a cached detector for the phoneme segment, using the cached detector for the phoneme segment; and When there is no cached detector for the phoneme segment, a detector is constructed for the phoneme segment.

2. The method of claim 1, wherein the new wake word detector is immediately constructed in response to receiving the verbal request.

3. The method according to claim 1, comprising: The user provides a verbal response to the natural language request, the verbal response confirming the new wake word, wherein the new wake word detector is constructed before the verbal response ends.

4. The method according to claim 1, wherein The verbal request is received at any time during a verbal conversation between the user and the device.

5. The method according to claim 1, further comprising: The new wake-up word detector is defined, wherein the new wake-up word detector is built locally on the device.

6. The method according to claim 1, further comprising: The new wake-up word detector is trained, wherein the training of the new wake-up word detector is performed remotely on a server connected to the device through a network.

7. The method according to claim 1, further comprising: The new wake word is added to a set of earlier wake words that includes at least earlier wake words, and the new wake word detector activates the device upon receiving a speech audio segment including the new wake word or a wake word from the set of earlier wake words.

8. The method according to claim 1, further comprising: One or more earlier wake-up words are replaced with the new wake-up word, so that after the replacement, the device is activated when a speech audio segment matching the new wake-up word is received, but is not activated when a speech audio segment matching the earlier wake-up word is received.

9. The method according to claim 8, further comprising the steps of: The wake-up word set of the device is later reset to include the earlier wake-up word, so that the new wake-up word detector activates the device upon receiving a speech audio segment including the earlier wake-up word.

10. The method according to claim 8, further comprising: The wake-up word set of the device is later reset to factory settings.

11. The method of claim 1 , wherein the new wake-up word detector is trained based on the speech audio segment, and the training is speaker-dependent such that when the new wake-up word is spoken by the user who created the new wake-up word, the device will be activated, and when the new wake-up word is spoken by someone else, the device will not be activated.

12. The method according to claim 11, further comprising: A model of the user's speech is generated for purposes of speaker verification.

13. A method according to claim 1, wherein the new wake-up word detector is trained based on the speech audio segment, and the training is speaker-independent so that the device will activate when the new wake-up word is spoken by the user who created the new wake-up word and other people who did not create the new wake-up word.

14. The method of claim 1, wherein the device solicits additional speech audio samples for the new wake-up word from the user, and trains the new wake-up word detector using the speech audio segments and the additional speech audio samples. The method of claim 14 , wherein the training of the new wake-word detector is performed locally on the device.

16. The method according to claim 1, wherein the step of constructing a new wake-up word detector comprises the following steps: The new wake-up word detector is trained using cached detectors for at least some portion of the phoneme segments of the new wake-up word.

17. The method of claim 1, wherein the new wake word detector is formed from phoneme segments streamed from a remote server and compared with phoneme segments of wake words stored on the device.

18. A method for modifying a set of one or more wake-up words for a voice-enabled device, comprising: receiving spoken words; parsing the utterance into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; Using automatic speech recognition to map the new wake-up word to a new wake-up word phoneme sequence; and The following steps are used to build a new wake-up word detector to identify the new wake-up word phoneme sequence as an activation trigger: Dividing the new wake-up word phoneme sequence into a sequence of two or more consecutive partial phoneme segments; For each partial phoneme segment, a corresponding partial wake-up word detector is provided; as well as Sequentially assembling the provided partial wake-up word detectors into the new wake-up word detector for the entire new wake-up word phoneme sequence, The step of providing a corresponding partial wake-up word detector for each partial phoneme segment includes: Check whether there is a cached detector for this part of the phoneme segment; When there is a cached detector for the portion of the phoneme segment, using the cached detector for the portion of the phoneme segment; and When there is no cached detector for the partial phoneme segment, a detector is constructed for the partial phoneme segment.

19. The method according to claim 18, wherein the step of dividing the new wake-up word phoneme sequence into phoneme segments comprises: The step of dividing the new wake-up word phoneme sequence into words.

20. The method according to claim 18, wherein the step of dividing the new wake-up word phoneme sequence into phoneme segments comprises: The step of dividing the new wake-up word phoneme sequence into separate syllables.

21. The method according to claim 18, wherein the step of dividing the new wake-up word phoneme sequence into phoneme segments comprises: The step of dividing the new wake-up word phoneme sequence into phoneme segments smaller than syllables.

22. The method of claim 18, wherein providing a partial wake-up word detector for a partial phoneme segment comprises: A dataset identifying audio segments labeled phonemically; searching the dataset to collect audio segments whose phoneme tags match the partial phoneme segments, and Based on the collected audio segments, the partial wake-up word detector is trained.

23. The method of claim 18, wherein providing a partial wake-up word detector for a partial phoneme segment comprises: Identifies the collection of cached wake word detectors indexed by the wake word; A cached wake-up word detector is retrieved for the partial phoneme segment.

24. The method according to claim 18, wherein the step of providing a corresponding partial wake-up word detector for each phoneme segment comprises the following steps: Partial wake-up word detectors for the continuous phoneme segments of the new wake-up word phoneme sequence are retrieved from a cache of the wake-up word detectors, and the partial wake-up word detectors are assembled into the new wake-up word detector.

25. The method of claim 18, further comprising: The new wake word is added to a set of earlier wake words that includes at least earlier wake words, and the new wake word detector activates the device upon receiving a speech audio segment including the new wake word or a wake word from the set of earlier wake words.

26. A method for modifying a set of one or more wake-up words for a voice-enabled device, comprising: receiving verbal requests; Parsing the spoken request into a natural language request and a speech audio segment, wherein the natural language request instructs the device to accept the speech audio segment as a new wake-up word; as well as The following steps are used to build a new wake-up word detector to identify the new wake-up word as an activation trigger: determining additional speech audio samples for the speech audio segment, converting the speech audio segment and the additional speech audio sample into a phoneme sequence, and Constructing the new wake-up word detector based on one or more phoneme sequences in the phoneme sequence, Among them, building a new wake-up word detector includes: dividing a phoneme sequence in the one or more phoneme sequences into a plurality of phoneme segments; and For each phoneme segment in the plurality of phoneme segments: Check whether there is a cached detector for the phoneme segment; When there is a cached detector for the phoneme segment, using the cached detector for the phoneme segment; and When there is no cached detector for the phoneme segment, a detector is constructed for the phoneme segment.

27. The method of claim 26, wherein the new wake word is a private wake word for the person who provided the verbal request, such that the device activates upon receiving the new wake word from the person and not others.

28. The method of claim 26, further comprising: Feedback is received from the user to verify the correctness of the phoneme sequence when two or more of the phoneme sequences are different from each other.

29. The method of claim 26, further comprising: The new wake word is added to a set of earlier wake words that includes at least earlier wake words, and the new wake word detector activates the device upon receiving a speech audio segment that includes the new wake word or a wake word from the set of earlier wake words.

30. The method of claim 26, further comprising: The earlier wake-up word is replaced by the new wake-up word, so that the wake-up word detector activates the device when receiving a speech audio segment including the new wake-up word, but does not activate the device when receiving a speech audio segment including the earlier wake-up word.

31. The method according to claim 26, further comprising the steps of: The wake-up word set of the device is reset to include the earlier wake-up word, so that the new wake-up word detector activates the device upon receiving a speech audio segment including the earlier wake-up word.

Images