Customizable keyword targeting system with keyword adaptation

Abstract

A user (202) can specify a wake word for a digital assistant (12) to trigger the digital assistant to respond to the wake word, with the user providing (204) one or more initial pronunciations of the wake word. The wake word can be unique, or at least not predetermined by a manufacturer or developer of the digital assistant's device. The one or more initial pronunciations of the wake word can then be augmented (214) with other potential pronunciations of the wake word that can be provided in the future, and those other potential pronunciations can be pruned (214) down to a threshold number of other potential pronunciations. One or more recordings of the one or more initial pronunciations of the wake word can then be used to train (218) a phoneme recognizer model to better recognize future instances of the wake word spoken by the user or another person using the initial pronunciations or other potential pronunciations.

Description

TECHNICAL FIELD

[0001] The present application relates generally to technologically innovative, non-conventional solutions that must stem from computer technology and produce a concrete technical improvement. In particular, the present application relates to the use of neural networks and other technologies to identify a wake word or other keyword to trigger a digital assistant response, where the keyword is not predetermined. BACKGROUND

[0002] Wake word detection is commonly used to trigger a digital assistant executing at a device to start listening for a task or information request spoken by a human and to which the assistant can respond. Assistants can be included in robots, smartphones, smart speakers, etc., and can be, for example, Amazon's Alexa®, Google's Assistant®, or Apple's Siri®. or Apple's Siri®.

[0003] However, a limitation of current technology is that the typically single, predefined, and generic wake word (often the name given to the assistant itself) lacks personalization. This limitation is intended to make the "keyword spotting" of the wake word as accurate as possible when attempting to detect the wake word in real-time speech, as current methods for keyword spotting generally require training a neural network for keyword spotting on thousands of training samples from numerous speakers saying the predetermined wake word to improve the accuracy of the assistant recognizing the wake word to a sufficient level. There is no adequate solution to the foregoing computer-related technical problem. SUMMARY

[0004] The present application recognizes that in a world where personalized robots, communication dialogue agents, and other types of digital assistants are growing exponentially, there is a need for a technical solution that can help users assign their own wake word name (or more generally, keyword) to an assistant and use that name to activate and converse with the assistant. The present application further recognizes that different users can interact with the same device / digital assistant at different times and can wish to each give the same assistant a different wake word or name.

[0005] Based on the foregoing, the present application further recognizes that customizable wake word / keyword spotting can be implemented and used in numerous applications without the need to train a neural network on thousands of training samples of different people saying the keyword.

[0006] Accordingly, in non-limiting examples consistent with the principles of the application, "customizable keyword spotting" can be considered a trainable keyword spotting method with the ability to add new keywords / wake words offline at the device locally without the aid of a server by registering a particular new keyword / wake word. The assistant can then begin to detect the newly added keyword / wake word again without the aid of a server in continuous real-time and "offline" speech to verify that the keyword has been spoken (although a server can be used for such purposes in some embodiments). The system / device containing the assistant can even adapt to the speaker during the registration process to make the system more familiar with the speaker's voice through a process of short-term keyword adaptation. This keyword adaptation can also be performed over shorter and longer term using the speaker's dialog with the assistant over time to improve the system's accuracy in detecting the speaker's wake word.

[0007] In more detail, in example embodiments, the phonemes spoken by a person can be detected, and the system can use a phoneme-keyword decoder to decode keywords spoken via phonemes to detect the keywords. To accomplish this, a phoneme recognizer model can be developed by first selecting a sufficient dataset for training and evaluating the model and applying it to keywords. For example, the TIMIT corpus can be used because it has sufficient annotated phoneme boundaries to train an efficient phoneme recognizer model, although it should be understood that a number of other corpora can be used. In accordance with the principles of the application, a recurrent neural network-based approach can be used to train the phoneme recognizer model, and in some examples, specifically a connectionist temporal classification (CTC)-based warping approach can be used.

[0008] Using this phoneme recognizer model, then, pronunciation augmentation and pruning can occur prior to the keyword adaptation described above to accurately decode possible keywords from phoneme sequences. To do this, the system with the phoneme recognizer model can be trained to approximately detect graphemes from phoneme sequences. In non-limiting examples, a phoneme can be one or more units of a phonetic system representing a single sound uttered by a user (e.g., "ch" in "Charlie"), and a grapheme can be one or more units of a writing system representing an individual phoneme. Thus, for example, given a particular phoneme sequence, one or more grapheme sequence representations of the most likely can be derived. A number of phoneme variants for each grapheme sequence can then be determined, with the sequitur tool being used to generate the first N phoneme variants (e.g., N = 5) of a grapheme sequence by means of a vocabulary / dictionary (e.g., approximately 80,000) to generate phoneme sequences that can be meaningful for a keyword, thereby mimicking possible variants of the phoneme sequence used to speak the keyword, as the user can not speak the exact same same keyword every time. Figure 14An example of this is shown in FIG. 14, where a single grapheme 1400 (CHARLIE) is fed into a sequitur G2P tool 1402 to render multiple phonemes 1404 (CH AA R L IY, CH AA R L AY, SH AA R L IY, etc.). Pruning can then be performed to reduce the number of possible variants of the phoneme sequence, thereby reducing processing time, while still maintaining the most accurate variants.

[0009] The phoneme sequence can then become the input and the grapheme sequence can become the output of the deep learning model training, where a model that can establish a phoneme recognizer can be trained using sequence-to-sequence training during adaptation. Once this model is trained, it can use the phonemes generated by the phoneme recognizer model and pass them to the sequence-to-sequence P2G model in order to subsequently decode the keyword as it is spoken in real-time by the user. Figure 15 An overview of this is shown in FIG. 15, where speech 1500 is fed into a phoneme recognizer model 1502 of a keyword decoder 1504 to ultimately decode a keyword spoken by a user after the keyword is registered.

[0010] Further, in accordance with the principles of the application, the system can detect keywords / wake words in two modes: an isolated keyword mode, where the keyword is spoken but no surrounding speech accompanies it to help identify that the system should be triggered; and a mode where the keyword is spoken around other pre-registered words to help identify that the system should be triggered. But to assist either mode, it should be understood that the system can incorporate keyword adaptation techniques in its customizable keyword positioning system (CKWS) architecture. This can involve the system investigating to try to improve keyword recognition accuracy by using short-term and long-term speaker characteristics.

[0011] Thus, in one aspect, a method for recognizing a keyword in speech includes detecting a first phoneme in the speech; converting the first phoneme to a first grapheme; adding a second grapheme from a lexicon to the first grapheme to render an augmented grapheme set; converting the augmented grapheme set to a second phoneme set; pruning the second phoneme set to render an output phoneme; and recognizing the keyword based at least in part on the output phoneme. The method can be performed using at least one neural network adapted to a human generating the speech, and the keyword in the speech can be indicated by a user and can not be pre-determined prior to being indicated in the speech.

[0012] In a non-limiting example, the lexicon can include base words and translated words. Each base word can be associated with “K” phoneme mappings, and each translated word can be associated with “N” phoneme mappings. Each base word can also be associated with “M” translated words. “K”, “N”, and “M” can be integers, and at least “K” can be greater than 1.

[0013] In some implementations, the method can include instructing the user to say the keyword a predetermined number of times greater than one, and further instructing the user that the keyword should contain at least three phonemes.

[0014] Also in some implementations, pruning the second set of phonemes can include removing duplicate phoneme sequences from the second set of phonemes. Pruning the second set of phonemes can also include eliminating a phoneme sequence from the second set of phonemes if the phoneme sequence is shorter than a threshold sequence length. Still further, pruning the second set of phonemes can include eliminating a phoneme sequence from the second set of phonemes if a first phoneme sequence in the second set of phonemes is not similar to a second phoneme sequence established by the first phoneme. Similarity can be measured using a fused distance metric based on Jaro Winkler normalized distance and Demerau Levenshtein normalized distance, such as Equation where a is a weighting parameter, where dl is a Demerau Levenshtein normalized distance, and where jw is a Jaro Winkler normalized distance.

[0015] Further, in some examples, pruning the second set of phonemes can include pruning based on sample matching of positive and negative samples, where a phoneme sequence from the second set of phonemes can be eliminated if it only matches negative samples. A phoneme sequence from the second set of phonemes that matches both positive and negative samples can not be eliminated from the second set of phonemes if a weighted average of the matching samples is greater than a threshold.

[0016] In some examples, the detecting can be performed using a phoneme recognizer model, and the method can further include, after the pruning, training the phoneme recognizer model based on the one or more recordings indicating the first phonemes. In these examples, the output phonemes can be first output phonemes, the phoneme recognizer model can be a first phoneme recognizer model, and the training can render a second phoneme recognizer model from the first phoneme recognizer model. The method can then include, after the training, rendering second output phonemes using the second phoneme recognizer model, and then either adding the second output phonemes to the first output phonemes to attempt to recognize the keyword based on both the first and second output phonemes, or replacing the first output phonemes with the second output phonemes to attempt to recognize the keyword based on the second output phonemes instead of the first output phonemes.

[0017] In another aspect, a method for identifying a keyword in speech includes detecting a first sequence of phonemes in the speech based on the keyword being spoken in a first instance, identifying a first sequence of graphemes using the first sequence of phonemes, and identifying at least second and third sequences of graphemes from a lexicon based on the first sequence of graphemes. The method also includes converting the second sequence of graphemes to at least second and third sequences of phonemes and converting the third sequence of graphemes to at least fourth and fifth sequences of phonemes. The method then also includes identifying the keyword being spoken again in a second instance using at least one of the second, third, fourth, and fifth sequences of phonemes, where the second instance occurs after the first instance.

[0018] Also, in some example implementations, the method can include eliminating the second sequence of phonemes and identifying the keyword being spoken again in the second instance using at least one of the third, fourth, and fifth sequences of phonemes but not the second sequence of phonemes.

[0019] Further, in some examples, the first sequence of phonemes can be detected using a first phoneme recognizer model, and the first sequence of phonemes in the speech can be stored as one or more recordings. In these examples, the method can include adapting the first phoneme recognizer model to the person providing the speech using the one or more recordings. The one or more recordings can be used to train the first phoneme recognizer model of the person to present a second phoneme recognizer model, where the second phoneme recognizer model can be used to identify the keyword being spoken in the second instance.

[0020] In yet another aspect, an apparatus includes at least one processor and at least one computer storage medium that is not a transitory signal. The computer storage medium includes instructions executable by the at least one processor to register a wake word with a digital assistant using a first phoneme recognizer model based on recordings of a person speaking the wake word at least in part by adding a first sequence of phonemes derived from the recordings to a lexicon accessible to the first phoneme recognizer model. The instructions are also executable to train the first phoneme recognizer model to present a second phoneme recognizer model using the recordings of the person speaking the wake word. The instructions are then executable to replace the first phoneme recognizer model with the second phoneme recognizer model, re-register the wake word with the digital assistant based on the recordings but using the second phoneme recognizer model, and update the lexicon by adding a second sequence of phonemes derived from the recordings using the second phoneme recognizer model.

[0021] In some examples, the recording can be a first recording, the person can be a first person, and the wake word can be a first wake word. In these examples, the instructions are further executable to register, using a second phoneme recognizer model, a second wake word to the digital assistant based on a second recording of a second person saying the second wake word at least in part by adding a third phoneme sequence to the lexicon. The third phoneme sequence can be derived from the second recording, where the second recording can be different from the first recording, the second person can be different from the first person, and the second wake word can be different from the first wake word. Also in these examples, the instructions are further executable to train the second phoneme recognizer model using the second recording to present a third phoneme recognizer model, replace the second phoneme recognizer model with the third phoneme recognizer model, re-register the second wake word to the digital assistant based on the second recording but using the third phoneme recognizer model, and update the lexicon by adding a fourth phoneme sequence derived from the second recording using the third phoneme recognizer model.

[0022] The details of the application, both as to its structure and operation, can best be understood with reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a block diagram of an example network of devices consistent with the principles of the application, where each device can include a keyword recognizer system as described herein;

[0024] Figure 2 and Figure 3 shows an architecture of a keyword spotting system according to the principles of the application;

[0025] Figure 4 shows example input / output phoneme and grapheme pairs for offline P2G model training according to the principles of the application;

[0026] Figure 5 illustrates actions taken during a first step of speech lexicon augmentation according to the principles of the application;

[0027] Figure 6 illustrates actions taken during a second step of speech lexicon augmentation according to the principles of the application;

[0028] Figure 7 shows examples of G2P and P2G variant files for various words / names according to the principles of the application;

[0029] Figure 8 and Figure 9 shows an example flowchart of actions taken during pronunciation augmentation and pruning based on P2G, G2P mapping according to the principles of the application;

[0030] Figure 10FIG. illustrates sample match during sound pronunciation pruning in accordance with the principles of the application;

[0031] Figure 11 FIG. illustrates an example flow diagram of keyword adaptation of a phoneme recognizer model in accordance with the principles of the application;

[0032] Figure 12 FIG. illustrates an example architecture of a phoneme recognizer model in accordance with the principles of the application;

[0033] FIG. 13 illustrates a flow diagram of example overall logic in accordance with the principles of the application;

[0034] Figure 14 FIG. 13 illustrates a flow diagram of example overall logic in accordance with the principles of the application;

[0035] Figure 15 is an example overview flow diagram indicating items for decoding keywords in accordance with the principles of the application. DETAILED DESCRIPTION

[0036] This disclosure relates generally to computer ecosystems including aspects of consumer electronics (CE) device networks, such as but not limited to, distributed computer game networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural networks and machine learning applications. The systems herein can include server and client components that are connected through a network connection such that data can be exchanged between the client and server components. The client components can include one or more computing devices including game consoles (such as Sony PlayStation®, Microsoft Xbox®, and / or Nintendo Wii® game consoles), virtual reality (VR) headsets, augmented reality (AR) headsets, game controllers, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers (such as laptop computers and tablet computers), and other mobile devices including smartphones and additional examples discussed below. These client devices can operate in a variety of operating environments. For example, some client computers can employ, for example, the Windows® operating system, the Windows® operating system from Apple®, or the Android® operating system, or operating systems produced by Samsung® or LG®. These operating environments can be used to execute one or more web browsers, such as the Internet Explorer® web browser from Microsoft®, the Safari® web browser from Apple®, or the Chrome® web browser from Google®. or by or or other manufacturers. These client devices can operate in a variety of operating environments. For example, some client computers can employ, for example, the Windows® operating system, the Windows® operating system from Apple®, or the Android® operating system, or operating systems produced by Samsung® or LG®. These operating environments can be used to execute one or more web browsers, such as the Internet Explorer® web browser from Microsoft®, the Safari® web browser from Apple®, or the Chrome® web browser from Google®. operating system, the Windows® operating system from Apple®, or the Android® operating system, or operating systems produced by Samsung® or LG®. These operating environments can be used to execute one or more web browsers, such as the Internet Explorer® web browser from Microsoft®, the Safari® web browser from Apple®, or the Chrome® web browser from Google®. operating system, the Windows® operating system from Apple®, or the Android® operating system, or operating systems produced by Samsung® or LG®. These operating environments can be used to execute one or more web browsers, such as the Internet Explorer® web browser from Microsoft®, the Safari® web browser from Apple®, or the Chrome® web browser from Google®. operating system, the Windows® operating system from Apple®, or the Android® operating system, or operating systems produced by Samsung® or LG®. These operating environments can be used to execute one or more web browsers, such as the Internet Explorer® web browser from Microsoft®, the Safari® web browser from Apple®, or the Chrome® web browser from Google®. or by or or or A browser manufactured or other browser program accessible to a website hosted by an Internet server discussed below. In addition, an operating environment according to the present principles can be used to execute one or more computer game programs according to the present principles.

[0037] The server and / or gateway can include one or more processors executing instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server can be connected through a local intranet or virtual private network. The server or controller can be instantiated by a game console (such as Sony PlayStation®, Microsoft Xbox®, or Nintendo Wii®), personal computer, etc.

[0038] Information can be exchanged between the client and server over a network. To this end and for security, the server and / or client can include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers can form an apparatus that implements a method of providing a secure community such as an online social website or video game website to network users for communication according to the present principles.

[0039] As used herein, instructions refer to computer-implemented steps for processing information in the system. Instructions can be implemented in software, firmware or hardware and include any type of programmed step implemented using processing components.

[0040] The processor can be any conventional general purpose single- or multi-chip processors that can execute a computer program. Such a processor can be implemented at least partially in hardware, which can include components such as a controller or various other conventional components such as a memory.

[0041] The software modules described by the flow charts and user interfaces herein can include various subroutines, programs, etc. Without limitation, logic stated as being performed by a particular module can be distributed over other software modules and / or combined with a single software module and / or made available in a shared library, etc. without limitation.

[0042] The present principles described herein can be implemented as hardware, software, firmware or combinations thereof; as such, the illustrative components, blocks, modules, circuits and steps are meant to be exemplary only and are implemented insofar as necessary.

[0043] ​The functions and methods described below, when implemented in software, can be written in an appropriate language such as, but not limited to, Java, C# or C++, and can be stored on or transmitted over a computer-readable storage medium such as a random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage such as digital versatile discs (DVD), magnetic disk storage or other magnetic storage devices including removable thumb drives, etc. A computer-readable medium can be connected to a computer. Such connections can include, for example, wired coaxial cables including fiber optic and twisted pair wires and digital subscriber line (DSL), and telephone lines. Such connections can include wireless communication connections, including infrared and wireless using Bluetooth technology, WiFi or other 802.11 technology, 3G and 4G networks, and other mobile communication technologies.

[0044] Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the drawings can be combined, interchanged or excluded from other embodiments.

[0045] A "system having at least one of A, B, and C" (likewise "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems that have: A only; B only; C only; both A and B; both A and C; both B and C; and / or A, B, and C.

[0046] Reference will now be made in detail to Figure 1, shows an example system 10 that can include one or more of the example devices mentioned above and further described below in accordance with the principles of the application. A first one of the example devices included in system 10 is a consumer electronics (CE) device such as an audio video device (AVD) 12, such as but not limited to an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls a TV). However, AVD 12 can instead be an appliance or a household item, e.g., a computerized Internet-enabled refrigerator, washer, or dryer. Alternatively, AVD 12 can also be a computerized Internet-enabled ("smart") phone, tablet computer, notebook computer, augmented reality (AR) headset, virtual reality (VR) headset, Internet-enabled or "smart" glasses, another type of wearable computerized device such as, e.g., a computerized Internet-enabled watch, a computerized Internet-enabled bracelet, other computerized Internet-enabled devices, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device such as an implantable skin device, etc. Regardless, it is to be understood that AVD 12 is configured to take the principles of the application (e.g., to communicate with other CE devices to take the principles of the application, to perform the logic described herein, and to perform any other functions and / or operations described herein).

[0047] Thus, to implement such principles, AVD 12 can be implemented by Figure 1Some or all of the illustrated components are established. For example, the AVD 12 can include one or more displays 14, which can be implemented by high definition or ultra-high definition ("4K") or higher flat panel screens, and can be touch-enabled for receiving user input signals by touch on the display. The AVD 12 can include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as, for example, an audio receiver / microphone, for inputting, for example, audible commands to the AVD 12 to control the AVD 12. The example AVD 12 can also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a wide area network, a local area network, etc., under the control of one or more processors 24, e.g., central processing units. A graphics processor 24A can also be included. Thus, the interface 20 can be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It is to be understood that the processor 24 controls the AVD 12 to implement the present principles, including the other elements of the AVD 12 described herein, such as, for example, controlling the display 14 to present images on the display and to receive input from the display. Further, it is noted that the network interface 20 can be, for example, a wired or wireless modem or router or other appropriate interface, such as a wireless telephony transceiver, or a Wi-Fi transceiver as mentioned above, etc.

[0048] In addition to the foregoing, the AVD 12 can also include one or more input ports 26, such as a high definition multimedia interface (HDMI) port or a USB port for physically connecting, e.g., using a wired connection, to another CE device, and / or a headphone port for connecting headphones to the AVD 12 for presenting audio from the AVD 12 to a user through the headphones. For example, the input port 26 can be connected, via wire or wirelessly, to a wired or satellite source 26a of audio video content. Thus, the source 26a can be, for example, a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a can be a game console or disc player containing content such as computer game software and databases. When implemented as a game console, the source 26a can include some or all of the components described below with respect to the CE device 44, and can implement some or all of the logic described herein.

[0049] The AVD 12 can also include one or more computer memories 28, such as disk-based storage or solid state storage, which in some cases are embodied as a separate device in the housing of the AVD, or as a personal video recording device (PVR) or video disk player inside or outside the housing of the AVD for playing back AV programs, or as a removable memory media, that are not transient signals. Also in some embodiments, the AVD 12 can include a location or position receiver, such as but not limited to a cell phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic position information, for example, from at least one satellite or cell tower, and to provide that information to the processor 24 and / or to determine, in conjunction with the processor 24, an altitude at which the AVD 12 is disposed. However, it should be understood that another suitable location receiver other than a cell phone receiver, a GPS receiver, and / or an altimeter can be used in accordance with the present principles, for example, to determine a position of the AVD 12 in all three dimensions.

[0050] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 can include one or more cameras 32, which can be, for example, thermal imaging cameras, digital cameras such as webcams, infrared (IR) cameras, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or video in accordance with the present principles.

[0051] Also included on the AVD 12 can be a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element can be a radio frequency identification (RFID) element. Other wireless communication elements can also be used, such as a ZigBee transceiver.

[0052] Still further, the AVD 12 can include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, gyro meters, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 can include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 can also include an infrared (IR) emitter and / or IR receiver and / or IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) can be provided for powering the AVD 12.

[0053] Still referring to Figure 1In addition to the AVD 12, the system 10 can include one or more other CE device types. In one example, a first CE device 44 can be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or via a server described below, while a second CE device 46 can include similar components as the first CE device 44. In the illustrated example, the second CE device 46 can be configured as a VR headset worn by a player 47 as shown, or a handheld game controller having one or more joysticks, one or more buttons, etc. that is manipulated by the player 47. In the illustrated example, only two CE devices 44, 46 are shown, it being understood that fewer or more devices can be used.

[0054] In the illustrated example, all three devices 12, 44, 46 can be members of an entertainment network, e.g., in a home / residence or distributed over several locations. Thus, unless explicitly stated otherwise, the present principles are not limited to the particular location shown by the dashed line 48.

[0055] The example non-limiting first CE device 44 can be established by any of the devices described above, e.g., a portable wireless laptop or notebook computer, a video game console or video game controller, and thus can have one or more of the components described below. The first CE device 44 can be a remote control (RC) for issuing AV play and pause commands to the AVD 12, or it can be a more complex device such as a tablet computer, a game controller that communicates with the AVD 12 via a wired or wireless link, a personal computer, a wireless telephone, etc.

[0056] Thus, the first CE device 44 can include one or more displays 50, which can be touch-enabled for receiving user input signals via touches on the display(s) 50. Additionally or alternatively, the display(s) 50 can be at least partially transparent displays configured for presenting AR and / or VR images, such as AR headset displays or "smart" glasses displays or "heads-up" displays, as well as VR headset displays or other displays.

[0057] The first CE device 44 can also include one or more speakers 52 for outputting audio in accordance with the present principles, and at least one additional input device 54, such as for example an audio receiver / microphone, for inputting audible commands to the first CE device 44, for example, to control the device 44. An exemplary first CE device 44 can also include one or more network interfaces 56 for communicating over the network 22 under control of one or more CE device processors 58. A graphics processor 58A can also be included. Thus, the interface 56 can be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including mesh network interfaces. It should be understood that the processor 58 controls the first CE device 44 to implement the present principles, including the other elements of the first CE device 44 described herein, such as for example controlling the display 50 to present images on the display and to receive input from the display. Further, it should be noted that the network interface 56 can be, for example, a wired or wireless modem or router, or other suitable interface, such as for example a wireless telephony transceiver, or a wireless network transceiver as described above, etc.

[0058] In addition to the foregoing, the first CE device 44 can also include one or more input ports 60 (such as for example HDMI ports or USB ports) for physically connecting (e.g., using a wired connection) to another CE device and / or an earphone port for connecting earphones to the first CE device 44 for presenting audio from the first CE device 44 to a user through the earphones. The first CE device 44 can also include one or more tangible computer readable storage media 62, such as disk-based or solid-state storage. Further in some embodiments, the first CE device 44 can include a location or positioning receiver such as but not limited to a cell phone and / or GPS receiver and / or altimeter 64 configured to receive geographic position information from at least one satellite and / or cell tower, for example, using triangulation, and to provide that information to the CE device processor 58 and / or to determine an altitude at which the first CE device 44 is disposed in conjunction with the CE device processor 58. However, it should be understood that another suitable location receiver other than a cell phone and / or GPS receiver and / or altimeter can be used in accordance with the present principles to determine a location of the first CE device 44 in, for example, all three dimensions, for example.

[0059] Continuing the description of the first CE device 44, in some embodiments, the first CE device 44 can include one or more cameras 66, which can be, for example, thermal imaging cameras, digital cameras such as webcams, and / or cameras integrated into the first CE device 44 and controllable by the CE device processor 58 to collect pictures / images and / or video, in accordance with the present principles. A Bluetooth transceiver 68 and other near field communication (NFC) elements 70 can also be included on the first CE device 44 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element can be a radio frequency identification (RFID) element.

[0060] Further, the first CE device 44 can include one or more auxiliary sensors 72 (e.g., motion sensors such as accelerometers, gyroscopes, odometry or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 can include other sensors that provide input to the CE device processor 58, such as one or more climate sensors 74 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 76. In addition to the foregoing, it is noted that, in some embodiments, the first CE device 44 can also include an infrared (IR) emitter and / or IR receiver and / or IR transceiver 78, such as an IR data association (IRDA) device. A battery (not shown) can be provided to power the first CE device 44, and / or the CE device 44 can be powered via a wired connection to a wall outlet. The CE device 44 can communicate with the AVD 12 through any of the aforementioned communication modes and related components.

[0061] The second CE device 46 can include some or all of the components shown for the CE device 44. One or both CE devices can be powered by one or more batteries and / or a wall outlet connection.

[0062] Reference is now made to the aforementioned at least one server 80, which includes at least one server processor 82, at least one tangible computer-readable storage medium 84 (such as a disk-based or solid-state storage device), and at least one network interface 86 that, under control of the server processor 82, permits communication with other devices on the network 22, and indeed can facilitate communication between servers and client devices in accordance with the present principles. It is noted that the network interface 86 can be, for example, a wired or wireless modem or router, Wi-Fi transceiver, or other appropriate interface such as, for example, a wireless telephony transceiver. Figure 1

[0063] ​Accordingly, in some embodiments, the server 80 can be an Internet server or a field of servers, and can include and perform "cloud" functionality, such that the devices of the system 10 can access a "cloud" environment via the server 80 in example embodiments such as a networked gaming application. Alternatively, the server 80 can be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG. 1. Figure 1

[0064] In addition to what has already been mentioned above, the logical blocks, modules, and circuits described below can be implemented or performed with general-purpose processors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs) or other programmable logic devices, such as application-specific integrated circuits (ASICs), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processors can be implemented by a controller or a combination of a state machine, or a combination of a computing device. Thus, the methods herein can be implemented as software instructions executed by a processor, a specially configured application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) module, or any other convenient means as would be understood by one of skill in the art. In the event that software instructions are employed, they can be embodied in a non-transitory device, such as a hard disk drive, CD ROM, or flash drive. The software code instructions can also be downloaded over the Internet.

[0065] As understood herein, machine learning can be a serialized process that involves identifying, acquiring, and preparing raw datasets, feature engineering, and generating extract, transform, and load (ETL) jobs that transform raw data into features for machine learning models, and designing and coding machine learning models. These models can be "trained" in computationally intensive tasks. The performance and latency of these models can even be evaluated.

[0066] Before continuing with the description of the other figures, it is to be understood that the present application contemplates two phases in accordance with the principles of the present application. Specifically, the present application contemplates both a registration phase / process and a detection phase / process. In the registration phase, the system can prompt (e.g., audibly via a speaker or visually via a display) the user to provide an audio sample of a keyword (e.g., specifically, a name) that they wish to use to trigger the assistant and respond to listen for subsequent commands from the user. The name provided by the user need not even be a pre-existing proprietary name, or even an existing word in the user's language (or any language for that matter).

[0067] ​Further, the system's prompt can instruct the user to not only say the user's desired name for the microphone accessible to the system to detect, but to say the desired name a threshold number of times greater than one, such as five times. The prompt can also instruct the user to use a keyword / name that is greater than a certain length, such as a name of at least three syllables or letters (thus can instruct at least three phonemes), as a name of less than three phonemes can impact the accuracy of future detection of the name, as it can lead to false detection of the name when in fact the user can not have attempted to trigger the system.

[0068] Further, the prompt can also instruct the user to not register by saying the name a predetermined number of times when the user has a physical condition that can adversely affect the clear pronunciation of the name, such as a cold or sore throat.

[0069] Assuming the user does not have a physical condition that affects the pronunciation of the user's desired name, the user can say the name a predetermined number of times when instructed by the system. The system can then record each pronunciation of the name based on the detection of the speech via the microphone. The system can then register the name / keyword by using the recorded speech waveforms to complete the registration phase without further input from the user.

[0070] Further, it should be appreciated that during registration, a simple consistency measure can be used to eliminate outliers, such as sounds from a user's pronunciation of the desired name that are significantly different from other instances of the user saying the desired name that can also be recorded during the same registration period. Further, only pronunciations that are longer than some minimum phoneme length and consistent with other enrollment samples (other instances of the user saying the name) can be considered valid.

[0071] In short, the second phase (the decoding phase) involves the subsequent real-time detection of the user's spoken name. This can sometimes be referred to as "decoding" and can trigger the assistant to "listen" for subsequent commands or requests from the user to act upon. The second phase uses a phoneme recognizer model trained during the first phase and initially established using a sequence-to-sequence model (e.g., trained using connectionist temporal classification) and the TIMIT corpus (and / or multiple other corpora) to detect phonemes from input speech. In the second phase, the detected phonemes can be decoded through a unique keyword decoder at the phoneme level (e.g., a keyword matcher using phoneme sequence similarity measures) to detect the spoken keyword.

[0072] Reference is now made to Figure 2 and Figure 3 which show the architecture of a keyword localization system according to the principles of the present application. Figure 2 showing an architecture that is simpler in form than Figure 3 in some aspects.

[0073] First from Figure 2At the outset, it illustrates both a registration phase 200 and a detection / decoding phase 210. The user 202 speaks the name "Tom" three times 204, which the system 206 detects to present a speech waveform 208. As shown, at the registration phase, the name "Tom" spoken by the user 202 is input into a phoneme recognizer artificial intelligence model 212, which outputs a phoneme sequence indicative of the phonemes recognized from the user speech, and provides the output to a pronunciation expansion and pruning model 214 for expanding and pruning the relevant phoneme sequence, which can be recognized at the detection phase 210 as indicative of the name "Tom." Pronunciation expansion and pruning will be discussed further below. Still referring to Figure 2 The pronunciation expansion and pruning model 214 can then adjust a keyword dictionary 216 based on the expansion and pruning, where a base version of the dictionary 216 is pre-selected by system developers and contains various grapheme entries and one or more corresponding pronunciation variants for each grapheme entry.

[0074] Thereafter, keyword adaptation 218 can be performed, where the phoneme recognizer model 212 can be trained using connectionist temporal classification (CTC)-based warping and the user speech 204 as training data to adapt the phoneme recognizer model 212 to the user speech. The phoneme recognizer model 212 can then be replaced with an adapted phoneme recognizer model 220 in order to subsequently decode the name when it is spoken and used as streaming audio at the detection phase 210 (although in some embodiments, the phoneme recognizer model 212 can also be used to decode at the detection phase). Keyword adaptation can include changing one or more phoneme recognizer models for the speaker's speech characteristics in order to improve keyword detection for that speaker. Thus, the keyword adaptation model 220 can be used to improve recognition accuracy of keyword detection, as shown in Figure 2 and Figure 3 and further adaptation can occur each time the system correctly recognizes "Tom" at the detection phase 210, such that additional training can occur over time as the user triggers the assistant.

[0075] The detection phase 210, as shown in Figure 2 is described in more detail, at the detection phase 210, the user 202 can thus again speak the name that the user provided to the system as a wake-up keyword, which can trigger the system to respond with an audio phrase 222 indicating that it has been triggered and is waiting for further input from the user of a task or command to execute. As shown in Figure 2As shown in the detection phase 210, the adapted phoneme recognizer 220 can be used to recognize that the user 202 said the phrase "Tom" using the keyword matcher / decoding module 224 to match the phonemes detected from the phrase "Tom" to entries in the lexicon 216 to present a decoded keyword 226 and trigger the system. The matching by the keyword matcher / decoding module 224 can be done using stream-adaptive matching; and / or using a distance metric to evaluate the distance between the keyword phoneme sequences stored in the lexicon 216 and the streamed phonemes output from the adapted phoneme recognizer 220 for the phrase "Tom" said by the user 202 to determine whether any entry in the lexicon 216 will output a match within a threshold distance.

[0076] Figure 3 In particular, it is shown how the intermittent keyword enrollment can also be used. However, before going into the details Figure 3 It should be appreciated that the intermittent enrollment can allow the user to enroll a new keyword pronunciation at a later time. This can happen if he / she is not satisfied with the wake-up response from a previously enrolled wake-up word, which can happen if the system is "too sensitive" and is triggered by a false positive when the user actually did not say the name he / she previously gave to the system, or if the system does not actually respond and wake up when the user says the keyword, which the user can realize.

[0077] The user can select the enrollment mode again at any time he / she desires to do intermittent enrollment (e.g., by providing a voice command when the system is successfully triggered based on input to a selector presented on a display to start the enrollment again, etc.). Additionally or alternatively, the system itself can trigger the intermittent enrollment if it determines that the user's voice has changed (e.g., due to no longer being as sick or having a sore throat as can have been the case when the initial enrollment was performed). The system itself can also trigger the intermittent enrollment if it determines that the user has asked the same question a threshold number of times greater than one within a threshold time of each other, where the question is recognized by the system each time it is provided even if the wake-up is triggered before or after the threshold number of times is reached. In any event, this new enrollment phase can occur as the initial enrollment did, which can require the user to say the preferred keyword again a valid threshold number of times.

[0078] Furthermore, it should be appreciated that during the intermittent enrollment, the user can even be allowed to provide a different name or pronunciation as the keyword for the system to wake up to, and after this different name is enrolled, it can be used to trigger the system to wake up in addition to the previously provided name which can still be used to trigger the system to wake up. However, it should be further appreciated that during the intermittent enrollment, the user can provide additional samples of the same wake-up word name as was provided during the initial enrollment.

[0079] Further, in some implementations, different users can have different registrations (one of which can be intermittent) so that different users can be assigned different names as respective wake words that can each trigger the same assistant / system, regardless of which user (or even another person) can subsequently speak any of the registered names to trigger the system. Thus, multiple different names / keywords can be assigned to a digital assistant for triggering the digital assistant to take action based on spoken commands.

[0080] Also during the intermittent registration, the user can be provided with the option to preserve prior registered instances (and append the newly registered pronunciations to the pronunciation dictionary) or to verify, check, and / or delete particular unwanted prior registered instances. This can be done through voice exchanges between the system and the user, via a graphical user interface (GUI) presented on a display, etc. For example, during registration verification in the intermittent registration, the user can simply say his / her keyword several times, and the system can show him / her any recognized matches to records (e.g., waveform files) from prior registrations. The user can then delete any registered instances from prior registrations that are returned as not matching the pronunciations provided at the start of the registration verification.

[0081] Now specifically describing Figure 3 During the intermittent registration 300, the user can say the desired name / keyword a threshold number of times as input 302. A phoneme recognizer model 304 that can or can not have been previously applied to one or more users can then use this input to present a pronunciation output 306 (e.g., "CHAH LIY") specified as P reg . The output 306 can then be provided as input to a pronunciation expansion and pruning model 308 to present an expanded set 310 of pronunciation outputs P aug that can be greater than 1. Both the pronunciation output 306 and the expanded set 310 can then be input into a dictionary 312.

[0082] Keyword adaptation 314 can then occur in which the phoneme recognizer model 304 can be retrained to present an adapted phoneme recognizer model 316 that can replace the phoneme recognizer model 304 and be used in a detection phase 318. Also during the adaptation 314, the same input 302 can be used to again present a pronunciation output, but using the model 316, which can then be provided as input to the pronunciation expansion and pruning model 308 to present another expanded set of pronunciation outputs. These pronunciation outputs can then be used in the detection phase 318 to determine an output pronunciation P test based on input 320 from the user from the adapted phoneme recognizer model 316. testThe pronunciation similarity measure 322 of the pronunciation output is determined to determine whether it is at least similar to the pronunciation output by a threshold amount T. As shown, if the similarity measure is greater than or equal to the threshold T, the keyword can be detected / recognized 324, or if the similarity measure is less than the threshold T, the keyword is not detected / recognized 326.

[0083] Now we describe pronunciation expansion in more detail, and in particular describe phonetic dictionary expansion as an aspect of pronunciation expansion, see Figure 4 . And again, it should be noted that the system can be based on phoneme recognition. Therefore, a phoneme recognizer model can use the user's input speech to detect phoneme sequences (which may be the building blocks of a speech vocabulary representation). In order for the system to recognize whether there are keywords in the user's speech, a phoneme to grapheme model can be used.

[0084] It should be understood that phonemes represent very basic information, and the system can be configured to predict many different possible pronunciations of a spoken keyword by training a neural network model using neural machine translation (NMT). The neural network model can learn to generate some or all possible grapheme combinations for speech input. This is referred to herein as phoneme-to-grapheme (P2G) model development.

[0085] To develop a P2G model, one or more neural translation techniques can be used. Figure 4 As shown in the example of , for a given word, there may be a variety of meaningful phonetic representations that represent the various possible ways of pronouncing the word. This itself may not always be a one-to-one mapping. But for Figure 4 Given a phoneme sequence listed as input 400 in , the system can identify one or more corresponding approximate / nearest grapheme sequences as output 402 to establish a one-to-one input / output pair, as shown.

[0086] Also like Figure 4 As shown, neural machine translation training 404 can be performed using these input / output pairs to develop a P2G model 406. The model may have been built by the system developer to take phoneme and grapheme pairs as input and target sequences. This developed phoneme-to-grapheme model can learn how to generate a grapheme sequence for a given phoneme sequence. The model can be trained offline at the device containing the system without the assistance of a server.

[0087] from Figure 4To continue and assist the reader in understanding the principles of the present application, a P2G system can take a phoneme sequence and generate a translated grapheme sequence based on the phoneme sequence. A G2P system can be considered the inverse of a P2G system and can use one or more systems from the text-to-speech domain. A G2P system can take a grapheme sequence (or word, in some embodiments) and generate N possible phonetic pronunciation variants for the given grapheme sequence (or word). Such a G2P system can be used to generate the input-output pairs shown in Figure 4 to build the P2G model / system 406.

[0088] Further, as used herein, a phnseq _phnrec may be a phoneme sequence generated by a phoneme recognizer model by feeding in a speech signal. As used herein, a grapheme _raw may be an original grapheme sequence (or word) derived from a phnseq _phnrec . A phnseq _g2p may be a phoneme sequence generated by a G2P system when the grapheme sequence is inputted into the G2P system. A grapheme _trans may be a translated grapheme sequence of a phoneme sequence.

[0089] It should be further understood that if a grapheme sequence (grapheme _raw ) is given a phoneme sequence (phnseq _g2p ) by a G2P system, and then that phoneme sequence is given through a P2G system, a translated grapheme sequence (grapheme _trans ) can be obtained that can or can not match the original grapheme sequence (grapheme _raw ). This is because, as this application acknowledges, such translations are not always absolute or perfect.

[0090] With the foregoing in mind, reference is now made to Figure 5 . In a first step, a relatively large list or dictionary 500 of graphemes (grapheme _raw ) and / or words and proper names can be collected, for example, by a system developer. For example, the dictionary 500 can be built from 139,000 graphemes. This dictionary 500 can then be passed through a G2P system 502 to generate phoneme sequence variants (phnseq _g2p ) 504, which in an example embodiment can be limited to a maximum of five phonetic variants per grapheme / word in the dictionary 500. However, another threshold number of variants other than five can be used in other examples, but in any case, a threshold number of variants can be used to reduce the variants to relax processing limitations on the system and allow it to operate efficiently. Figure 5Also shown in FIG. 5 are example names / graphemes 506 from the dictionary, along with phonetic variants 508 for each name / grapheme 506 .

[0091] Then in Figure 6 In the second step shown, each obtained phoneme sequence variant 504 may be passed through the P2G system 600 to generate the nearest translated grapheme sequence / representation variant 602 for each phoneme sequence variant 504 passed through the P2G system 600. Figure 6 It will also be appreciated that the example phonetic sequence variants 604 may be mapped to one or more example translated grapheme sequence / representation variants 606 .

[0092] In the third step, a combined representation of steps one and two can be generated, such as Figure 7 As shown. Figure 7 In the example shown, the file can contain both G2P and P2G variants of all graphemes / words in a unified representation. Thus, the combined representation can include grapheme _raw The first column of the entry is 700, Phnseq _g2p The second column of the entry 702 and grapheme _trans A third column 704 of entries, where entries in each row map to entries in other columns of the same row.

[0093] Thereafter, in a fourth step, P2G mapper development can be performed, assuming that one of the purposes of these steps may be to find the most appropriate phonetic substitute for the word spoken during enrollment in order to improve detection accuracy. Due to the potential non-one-to-one mapping of original graphemes to translated graphemes, words / graphemes that sound similar but different may be _raw May be translated into some of the same grapheme _trans For example, a grapheme like ["SHELLEY", "SHELLY"] might be translated as "SH ELLY". Therefore, one of the phoneme sequence variants of "Shelley" might be the same as the one translated as "Shelly". In order to centrally represent such similar-sounding graphemes (or words), a dictionary can be constructed in this fourth step, which gives as a return value the given translation grapheme. _trans All similar-sounding graphemes _raw probability. For example, "SHELLY": ["SHELLEY", "SHELLY"], and "PORTA": ["PUERTO", "PORTA", "PORTO"].

[0094] Thus, given the above steps and the sequence of transliterated graphemes, it can be determined that the graphemes (or words) that sound similar _raw and thus all phonetic variants associated with those graphemes (or words) can be determined. Thus, a centralized dictionary of graphemes _raw to phonetic variants and graphemes _trans to similar graphemes _raw may be built. In some examples, this centralized dictionary can be referred to as a bidirectional P2G, G2P mapping dictionary.

[0095] Still describing pronunciation augmentation, but now specifically with reference to the pronunciation augmentation using P2G, G2P mapping according to the above disclosure, it should be understood that in the enrollment phase, the system can receive from the user (as described above) the five spoken pronunciations of the keyword. However, in understanding that this can be a somewhat limited representation of the user-selected keyword relative to, for example, a typical speech-to-text model that can have been trained on hundreds or thousands of samples from many people, it can be possible that issues such as repeatedly failing to recognize the user’s keyword can arise if only these five samples from the user are used in the detection phase to identify the user’s keyword when spoken again. And since the user-selected keyword can be novel or unique, thousands of samples can not be available as can be the case for a developer- or manufacturer-predefined keyword, the present application recognizes the advantage of generating many possible “meaningful variants” of the user’s pronunciation of the keyword in order to help the system better model and detect the user’s preferred keyword.

[0096] Thus, reference is made to Figure 8 to help understand the augmentation of phonetic variants at a “higher level.” As Figure 8As shown, the five registered pronunciations 800 from the user can be provided to a phoneme recognizer model 802 to output five corresponding phoneme sequences 804. These sequences 804 are then provided to a P2G translation model 806 for the system to determine at diamond 808 whether any grapheme translations from the P2G translation model 806 correspond to the predetermined grapheme / name / word in the keyword dictionary at least within a threshold level of similarity, such as eighty percent (e.g., if the actual keyword spoken by the user is a unique, made-up word, but still similar to an entry in the dictionary). If no grapheme translations correspond to the predetermined grapheme / name / word in the keyword dictionary, then only the phonetic sequences 804 of the five registered pronunciations can be used for keyword spotting during the detection phase. But assuming at least one grapheme translation corresponds to the predetermined name or other word in the keyword dictionary, a bidirectional P2G, G2P mapping dictionary 810 as described herein can be accessed to determine at diamond 812 whether there are any alternative pronunciations, and in response to an affirmative determination at diamond 812, the keyword dictionary can be augmented with the alternative pronunciations at step 814 to establish a registered keyword phonetic dictionary 816. From there, duplicate phonetic sequences can be removed from the registered keyword phonetic dictionary 816 to present a final registered keyword phonetic dictionary 818 that can be used in the decoding phase.

[0097] Figure 9 This is also illustrated. The Phnseq _phnrec 900 (specified as P r in Figure 9 ) can be provided as input to a P2G model 902 to output grapheme _trans output 904 (specified as W _raw in Figure 9 ). W _raw can then be provided as input to a grapheme augmentation component / mechanism 906 that uses a P2G, G2P dictionary to present grapheme m output 908 based on the grapheme _g2p input 904. In some examples, the component / mechanism 906 can be comprised of a neural network model, a text file, and / or an algorithm.

[0098] The grapheme _g2p output 908 is listed as W1 through W _raw in Figure 9 , which is then provided as input to a phonetic sequence augmentation model 910 to generate phonetic sequences phnseq 11 912 using a P2G, G2P dictionary, where multiple corresponding phonetic sequences phnseq 1n1 correspond to one grapheme_raw (e.g., P 11 -P 1n1 corresponding to W1, and P 21 -P 2n2 corresponding to W2). Thereafter, the phonetic sequence phnseq _g2p may undergo a pruning process 914, which will be described further below, resulting in a final augmented phonetic sequence (P1-P n ) 916, which is then added to the keyword phonetic dictionary.

[0099] Thus, to achieve improved keyword coverage, a trained P2G model can be used to augment the registered sequences with more phonetic variants. It can now be appreciated that this process can be used to generate phonetic variants to have the “best” phoneme sequence representation of a registered keyword in the keyword / wake-word dictionary used in the detection phase. It should further be appreciated that the principles of the present invention can be extended to various languages and platforms, and help reduce computational complexity as well as improve detection accuracy.

[0100] Now describing in detail the pronunciation pruning as referenced herein, there can be at least three types of pruning. Optionally, the three types described below can be executed in the order set forth below. But first, it should be appreciated that one of the reasons for pronunciation augmentation is to cover reasonable differences of the same keyword, as in the detection phase, the same keyword can “sound” slightly different compared to the pronunciation at registration, because of the speaker’s physical condition (e.g., sore throat), channel differences, the user just spoke the keyword slightly different than during registration, etc. It should further be appreciated that one of the reasons for pronunciation pruning is to keep “good” augmented pronunciations to improve detection accuracy and to remove “bad” augmented pronunciations to reduce false positives / false activations of the assistant system.

[0101] For two of the three types of pruning described above, they can be considered as rule-based pronunciation pruning, which can be collectively considered as a “part” of the pruning. In any case, the first type of pruning belonging to rule-based pronunciation pruning eliminates augmented pronunciations that are shorter than a required or minimum phoneme length. The second type of pruning belonging to rule-based pronunciation pruning eliminates augmented pronunciations that are not sufficiently similar to the original decoded pronunciation P a from which the various augmented pronunciations P r were derived.

[0102] The first type of pruning only involves identifying augmented pronunciations that are shorter than a required or minimum threshold phoneme length (e.g., set by system developers). They can be eliminated and thus not added to the keyword dictionary. Thus, through this type of pruning, augmented pronunciations that are too short can be filtered out.

[0103] The second type of pruning involves measuring P r against Pa similarity between them. A fusion distance measure based on Jaro Winkler normalized distance and Demerau Levenshtein normalized distance based on average can be used. The fusion distance measure equation can be where a is a weighting parameter, where dl is the Demerau Levenshtein normalized distance, and where jw is the Jaro Winkler normalized distance. Thus, with this type of pruning, augmented pronunciations that are too "far" (not similar enough) from the original decoded pronunciation can be filtered out if the resulting fusion distance is greater than a threshold distance (e.g., determined by system developers).

[0104] A third type of pruning can be considered as sample-based pronunciation pruning. Here, pronunciation pruning can be based on using positive and negative samples to be "deeper" than rule-based pruning. It involves preparing a positive data sample set (based on the keyword itself) and a negative data sample set (non-keywords) to obtain a sample match score associated with matching pronunciations.

[0105] The positive samples can be established by the user recording the keyword during the first part of the registration phase and using a phoneme recognizer model to decode the keyword recording (e.g., a waveform file), while the negative samples can be selected from a subset of a word database collected offline and determined by system developers. Optionally, these negative samples can sound very different from the keyword such that they would not be considered as positive samples, but still not too different from the keyword such that they are in the vicinity of the keyword in the pronunciation space and are likely to cause confusion. The negative data sample set can be extracted by pronunciation distance measurement.

[0106] The sample match can thus be understood as a process of obtaining the most similar pronunciation P a to the decoded pronunciation of the sample P0. The similarity / distance measure can be Dist(P a , P0) = min(P a , P0), where P a can be various augmented pronunciations from P0.

[0107] The positive / negative sample decoded pronunciation pruning process can be as follows: Figure 10The augmented pronunciation 1000 (designated by the circle with the smaller radius) is shown as matching a negative sample 1004 if, after rule-based pruning, the augmented pronunciation 1000 is most similar to the decoded pronunciation of one or more negative samples. If the augmented pronunciation 1000 matches a positive sample 1006 if, after rule-based pruning, the augmented pronunciation 1000 is most similar to the decoded pronunciation of one or more positive samples, then the pronunciation can be retained and ultimately added to the keyword dictionary. Also, for clarity, note that the augmented pronunciation 1002, prior to rule-based pruning, is designated by the circle with the larger radius, as shown in FIG. 10, and the original decoded pronunciation P0 1008 is typically shown at the center of these overlapping circles 1000, 1002. Figure 10

[0108] However, it should also be noted that in some scenarios, the same augmented pronunciation can match multiple samples, some of which can be positive samples and some of which can be negative samples. In such a case, a weighted average score of the matching samples of the same augmented pronunciation can be used, and the augmented pronunciation can be retained if the weighted average of the matching positive / negative samples is greater than a weighted average threshold.

[0109] For example, the weight can be negative if the match is a negative sample, and positive if the match is a positive sample, but the sample weight being positive or negative can be intentionally ignored when using a distance metric for determining distance. The distance can then be used to determine the weight size, e.g., the smaller the distance, the larger the weight, and vice versa. Thus, it can be appreciated that if a match occurs with a negative sample that is close enough in distance, then the result can be a relatively large “negative” weight, but if the weighted average is still greater than the threshold, then the augmented pronunciation can still be retained.

[0110] Keyword adaptation will now be described, in which a system containing an assistant can adapt to a user providing pronunciations of desired keywords in a registration phase to make the system more familiar with the user’s voice. In particular, keyword adaptation can be performed to train a generic phoneme recognizer model (possibly initially used in the registration phase) using connectionist temporal classification (CTC) warping and the user’s newly registered keyword recordings as training data to specialize the phoneme recognizer model to the user’s speech to further improve accurate keyword localization / decoding, and thus improve the user experience and interaction with the assistant / system.

[0111] Figure 11 A keyword adaptation process in the registration phase is shown in accordance with the principles of the application. In a non-limiting implementation, keyword adaptation can involve five steps. First, a “regular” registration 1100 from input keyword recordings 1102 to a pronunciation dictionary 1104 can be performed as described above, just as if there were no keyword adaptation. This can involve using an original or generic phoneme recognizer model 1106 to render the input keyword recordings 1102 into a set of phonemes 1108, as described above.​reg (e.g., "CH AH L IY") can then be provided as input to a pronunciation expansion and pruning model 1108 to render an expanded set 1109 of pronunciations P that can be greater than 1. Both the pronunciation output 1107 and the expanded set 1109 can be input into the lexicon 1104. aug

[0112] Second, the keyword-pruned phoneme recognizer model can be retrained / adapted using the keyword-pronunciation pairs as training samples in a training / retraining phase 1110. Further, it should be understood that after pronunciation expansion and pruning 1108, some recordings can have multiple pronunciations associated with them, in which case the recordings can be duplicated / reused to create a different recording-pronunciation pair for each pronunciation after pronunciation expansion and pruning.

[0113] Third, after training / retraining, the keyword-adapted phoneme recognizer model 1116 can replace the original phoneme recognizer model 1106, generally as indicated by arrow 1112.

[0114] Fourth, the keyword pronunciations can be re-registered in a re-registration phase 1114, as the system can go through the same registration phase, including phoneme recognition 1118 and pronunciation expansion and pruning 1120, but using the updated, keyword-adapted phoneme recognizer 1116.

[0115] Fifth, the keyword pronunciation lexicon 1104 can be updated as specified by block 1124, as the expanded previous pronunciation lexicon 1104 can be replaced with the new pronunciation lexicon 1122 generated based on the steps in paragraph four above. The previous pronunciation lexicon 1104 can then be deleted, or it can be retained as a backup in the system-accessible history / storage in case the new pronunciation lexicon 1122 is ultimately proven to be not useful (e.g., if it was generated based on audio samples of the user when the user had a sore throat or other illness affecting speech) so that it can be used again at a later time. In other examples, however, the new lexicon 1122 (or at least the new entries of the lexicon 1122) can be appended to the previous keyword pronunciation lexicon 1104 that was expanded prior to keyword adaptation.

[0116] ​Further, it should be appreciated that in some examples, in which one or more users can assign multiple keywords / names to a single assistant / system, the above steps can be repeated for keyword adaptation. Thus, keyword adaptation according to the above steps can be performed using records of one of the keywords assigned to the assistant (from the same person and / or different people’s records), and then the modified model can be modified again using records of a different one of the keywords assigned to the assistant. For each set of records corresponding to a different keyword assigned to the assistant, and even for the same keyword, if users intermittently register as described above to provide new sets of records for keywords that the user has previously provided, adaptation according to these steps can continue. This can help the system to recognize any person who speaks any of the registered keywords of the assistant / system.

[0117] Figure 12 An example phoneme recognizer model architecture is shown that is established by a recurrent neural network (RNN) operating according to principles of the present application to receive a speech sequence 1200 as input and output one or more phoneme probabilities 1202 as output. As shown, the input 1200 can be passed to an input layer, then through a number of long short-term memory (“LSTM”) and batch normalization (“batch norm”) hidden layers 1204, and one or more fully connected and rectified linear unit (ReLU) layers 1206 and one or more fully connected and softmax function layers 1208, ultimately providing the output 1202 through an output layer.

[0118] From Figure 12 Continuing, it should be appreciated that another feature can also be used according to principles of the present application. Specifically, at the decoding stage, pre-registered keyword spotting can also be used for words that can accompany or surround a user-specified and potentially unique keyword. A separate dictionary of contextually relevant pre-registered keywords can be used for such purposes to further enhance the system’s ability to distinguish between a user’s actual attempt to trigger the system to wake up and a false positive in which the user did not intend to do so. Thus, for example, if the system identifies a user’s unique keyword that has been provided within a certain probability, but that probability does not reach the probability threshold for waking up / activating the assistant based on the unique keyword alone, the system can consider the surrounding pre-registered keywords to determine whether they contain an information request or command or other item that the system can execute, or rather whether they contain only speech that was not intended to be input into the system. For example, “Tom, turn on the TV” can be an example of the former (where “Tom” is the wake-up keyword intended by the user), while “Tom, my favorite color is green” can be an example of the latter. Thus, the first phrase can trigger the system to wake up / activate, while the second phrase can not.

[0119] FIG. 13 is now described, which illustrates a flowchart of example overall logic that can be performed by a digital assistant system in accordance with the principles set forth herein. Beginning at block 1300, the system can prompt a user to say a desired keyword / wakeword a predetermined number of times, and can even prompt that the keyword should be at least a minimum length. At block 1302, the device can detect and record a first phoneme in the user’s speech that indicates a non-predetermined keyword of the user. Thereafter, at block 1304, the system can convert the first phoneme to a first grapheme, and then at block 1306, add a second grapheme from a grapheme dictionary using the first grapheme to present an augmented grapheme set.

[0120] The logic can then proceed from block 1306 to block 1308, where the system can convert the augmented grapheme set to a second phoneme set. Thereafter, at block 1310, the system can prune the second phoneme set by eliminating, for example, phoneme sequences that are repeated, shorter than a threshold, not sufficiently similar, and / or have already been matched to a negative sample to present an output phoneme. Also at block 1310, the output phoneme can be added to a pronunciation dictionary.

[0121] The logic can continue from block 1310 to block 1312, where the system can train a phoneme recognizer model (used at block 1302 to detect phonemes) based on the recordings (also from block 1302) using a connectionist temporal classification (CTC)-based warping to present a trained phoneme recognizer model that can replace the initial phoneme recognizer model (used at block 1302) to re-register the keyword of the user. The logic can then move to block 1314, where the system can add additional output phonemes to the pronunciation dictionary based on output from the trained phoneme recognizer model during re-registration. The logic can continue from block 1314 to block 1316, where, during a detection phase, the system can use output from the trained phoneme recognizer model and use the pronunciation dictionary with added pronunciations to recognize the keyword spoken by the user or another person. The logic can then move to block 1318, if needed, where the system can repeat steps 1300-1314 for a different keyword or the same keyword, for example, during an intermittent registration.

[0122] The logic can continue from block 1314 to block 1316, where, during a detection phase, the system can use output from the trained phoneme recognizer model and use the pronunciation dictionary with added pronunciations to recognize the keyword spoken by the user or another person. The logic can then move to block 1318, if needed, where the system can repeat steps 1300-1314 for a different keyword or the same keyword, for example, during an intermittent registration.

[0123] It will be appreciated that, although the principles of the application have been described in reference to some example implementations, these implementations are not intended to be limiting, and that various alternatives and modifications can be used to implement the subject matter recited in the claims.

Claims

1. A method for identifying a keyword in speech, comprising: detecting first phonemes in the speech; converting the first phonemes to first graphemes; adding second graphemes from a lexicon to the first graphemes to present an augmented grapheme set; converting the augmented grapheme set to a second phoneme set; pruning the second phoneme set to present output phonemes; and identifying the keyword based at least in part on the output phonemes.

2. The method of claim 1, comprising performing the method using at least one neural network adapted to a human who generated the speech. base words, each base word related to "K" phoneme mappings; translated words, each translated word related to "N" phoneme mappings; and each base word related to "M" translated words, where "K", "N", and "M" are integers and at least "K" is greater than 1.

3. The method of claim 1, wherein the dictionary comprises:

4. The method of claim 1, wherein the keyword in the speech is indicated by a user, the keyword not predetermined prior to the keyword being indicated in the speech.

5. The method of claim 1, comprising: indicating to a user to say the keyword a predetermined number of times greater than 1, and indicating to the user that the keyword should contain at least three phonemes.

6. The method of claim 1, wherein pruning the second phoneme set comprises removing duplicate phoneme sequences from the second phoneme set.

7. The method of claim 1, wherein pruning the second phoneme set comprises eliminating phoneme sequences from the second phoneme set if a particular phoneme sequence is shorter than a threshold sequence length.

8. The method of claim 1, wherein pruning the second phoneme set comprises eliminating phoneme sequences from the second phoneme set if a first phoneme sequence in the second phoneme set is not similar to a second phoneme sequence established from the first phonemes.

9. The method of claim 8, wherein similarity is measured using a fusion distance metric based on Jaro Winkler normalized distance and Demerau Levenshtein normalized distance.

11. The method of claim 1, wherein pruning the second phoneme set comprises pruning based on sample matching of positive samples and negative samples, wherein a phoneme sequence from the second phoneme set is eliminated if it only matches a negative sample.

10. The method of claim 9, wherein the fusion distance metric is determined using the equation where a is a weighting parameter, where di is a Demerau Levenshtein normalized distance, and where jw is a Jaro Winkler normalized distance.

12. The method of claim 11, wherein a phoneme sequence from the second phoneme set that matches a positive sample and a negative sample is not eliminated from the second phoneme set if a weighted average of matching samples is greater than a threshold value.

13. The method of claim 1, wherein the detecting is performed using a phoneme recognizer model, and wherein the method comprises: training the phoneme recognizer model based on one or more recordings indicating the first phonemes after the pruning.

14. The method of claim 13, wherein the output phonemes are first output phonemes, wherein the phoneme recognizer model is a first phoneme recognizer model, wherein the training presents a second phoneme recognizer model from the first phoneme recognizer model, and wherein the method comprises: ​ after the training, using the second phoneme recognizer model to present second output phonemes; and either adding the second output phonemes to the first output phonemes to attempt to recognize the keyword based on both the first and second output phonemes, or replacing the first output phonemes with the second output phonemes to attempt to recognize the keyword based on the second output phonemes instead of the first output phonemes.

15. A method for recognizing a keyword in speech, comprising: detecting a first sequence of phonemes in the speech based on the keyword spoken in a first instance; recognizing a first sequence of graphemes using the first sequence of phonemes; recognizing at least second and third sequences of graphemes from a dictionary based on the first sequence of graphemes; converting the second sequence of graphemes into at least second and third sequences of phonemes and converting the third sequence of graphemes into at least fourth and fifth sequences of phonemes; and recognizing the keyword spoken again in a second instance using at least one of the second, third, fourth, and fifth sequences of phonemes, the second instance occurring after the first instance.

16. The method of claim 15, further comprising: eliminating the second sequence of phonemes; and recognizing the keyword spoken again in the second instance using at least one of the third, fourth, and fifth sequences of phonemes but not using the second sequence of phonemes.

17. The method of claim 15, wherein the first sequence of phonemes is detected using a first phoneme recognizer model, wherein the first sequence of phonemes in the speech is stored as one or more recordings, and wherein the method comprises: using the one or more recordings to adapt the first phoneme recognizer model to a person who provided the speech.

18. The method of claim 17, wherein the one or more recordings are used to adapt the first phoneme recognizer model to the person to present a second phoneme recognizer model, the second phoneme recognizer model used to recognize the keyword spoken in the second instance.

Images