Customizable keyword positioning system with keyword adaptation

Abstract

A user (202) may specify a wake-up word for a digital assistant (12) to trigger the digital assistant to respond to the wake-up word, where the user provides (204) one or more initial pronunciations of the wake-up word. The wake-up word may be unique, or at least not predetermined by a device manufacturer or developer of the digital assistant. One or more initial pronunciations of the keyword may then be augmented (214) with other potential pronunciations of the wake-up word that may be provided in the future, and those other potential pronunciations may be pruned (214) down to a threshold number of other potential pronunciations. A phoneme recognizer model may then be trained (218) using one or more records of one or more initial pronunciations of the wake-up word to better recognize future instances of the wake-up word spoken by the user or another person using the initial pronunciation or other potential pronunciation.

Description

[0001] This patent application is a divisional application of the following invention patent application:

[0002] Application Number: 202080071838.3

[0003] Application date: August 26, 2020

[0004] Invention Title: Customizable Keyword Targeting System with Keyword Adaptation Technical Field

[0005] This application generally relates to unconventional solutions that are technologically innovative, stemming from computer technology and resulting in specific technological improvements. In particular, this application relates to using neural networks and other technologies to identify wake words or other keywords to trigger a digital assistant response, where the keywords are not predetermined. Background Technology

[0006] Wake word detection is typically used to trigger a digital assistant running on a device to begin listening to a task or information request verbally provided by a human and in which the assistant may respond. Assistants can be included in robots, smartphones, smart speakers, etc., and can be, for example, Amazon's... Or Apple's .

[0007] However, current technology is limited by the fact that wake words, typically singular, predefined, and generic (usually the assistant's name itself), lack personalization. This limitation aims to make "keyword localization" of wake words as accurate as possible when attempting to detect wake words in real-time speech, as current methods for keyword localization often require training neural networks on thousands of training samples from numerous speakers uttering the predetermined wake word to improve the assistant's accuracy in recognizing wake words to a sufficient level. Currently, there is no suitable solution to the aforementioned computer-related technical problems. Summary of the Invention

[0008] This application recognizes the need for a technological solution in a world of exponential growth in personalized bots, conversational agents, and other types of digital assistants. This solution would allow users to assign their own wake word names (or more generally, keywords) to their assistants and use those names to activate and converse with them. This application also recognizes that different users may interact with the same device / digital assistant at different times and may wish to give the same assistant different wake words or names.

[0009] Based on the foregoing, this application further recognizes that customizable wake word / keyword localization can be implemented and used in numerous applications without training neural networks on thousands of training samples from different people who say the keywords.

[0010] Therefore, in a non-limiting example consistent with the principles of the invention, "customizable keyword localization" can be considered a trainable keyword localization method that has the ability to add new keywords / wake words by registering specific new keywords / wake words offline locally on the device without the help of a server. An assistant can then begin detecting the newly added keywords / wake words again in continuous real-time and "offline" speech without server assistance to verify that the keywords have been spoken (although in some embodiments a server may be used for such purposes). The system / device incorporating the assistant can even be adapted to the speaker during the registration process to familiarize the system with the speaker's voice through a short-term keyword adaptation process. This keyword adaptation can also be performed over time using conversations between the speaker and the assistant in both short-term and long-term applications to improve the accuracy of the system in detecting speaker wake words.

[0011] More specifically, in the example implementation, spoken phonemes can be detected, and the system can use a phoneme-keyword decoder to decode spoken keywords via phonemes to detect keywords. To achieve this, a phoneme recognizer model can be developed by first selecting a sufficient dataset for training and evaluating the model and then applying it to keywords. For example, the TIMIT corpus can be used because it has sufficient annotated phoneme boundaries to train an efficient phoneme recognizer model, but it should be understood that several other corpora can be used. According to the principles of the invention, a recurrent neural network-based approach can be used to train the phoneme recognizer model, and in some examples, a warping method based on connectionist temporal classification (CTC) can be specifically used.

[0012] Using this phoneme recognizer model, pronunciation augmentation and pruning can then occur before the aforementioned keyword adaptation to accurately decode possible keywords from the phoneme sequence. To do this, a system with a phoneme recognizer model can be trained to approximately detect epigraphs from the phoneme sequence. In a non-restrictive example, a phoneme can be one or more units representing a single sound produced by a user in a speech system (e.g., the "ch" in "Charlie"), and an epigraph can be one or more units representing a single phoneme in a writing system. Thus, for example, given a particular phoneme sequence, the most probable epigraph sequence representation can be derived. Many phoneme variants of each epigraph sequence can then be identified, where the top N phoneme variants of the epigraph sequence (e.g., N=5) are generated using a sequitur tool with the aid of a vocabulary / dictionary (e.g., approximately 80,000), generating potentially meaningful phoneme sequences for keywords, thus mimicking possible variants of the phoneme sequence used to pronounce the keyword, since a user may not pronounce the exact same keyword every time. Figure 14An example of this scenario is shown, where a single character 1400 (CHARLIE) is fed into a sequitur G2P tool 1402 to represent multiple phonemes 1404 (CH AA RL IY, CH AA RL AY, SH AA RL IY, etc.). Trimming can then be performed to reduce the number of possible variations of the phoneme sequence, thereby reducing processing time while still preserving the most accurate variations.

[0013] The phoneme sequence can then be used as input and the character sequence as output for training a deep learning model, where sequence-to-sequence training can be used during adaptation to train a model that can build a phoneme recognizer. Once the model is trained, it can use the phonemes generated by the phoneme recognizer model and pass them to the sequence-to-sequence P2G model for subsequent decoding as the user speaks keywords in real time. Figure 15 An overview of this is shown in the figure, in which speech 1500 is fed to the phoneme recognizer model 1502 of the keyword decoder 1504 to ultimately decode the keywords spoken by the user after registering the keywords.

[0014] Furthermore, according to the principles of the invention, the system can detect keywords / wake words in two modes: an isolated keyword mode, where the keyword is spoken without any surrounding speech to aid in identification and should trigger the system; and a mode where the keyword is spoken around other pre-registered words to aid in identification and should trigger the system. However, to assist in either mode, it should be understood that the system can incorporate keyword adaptation techniques within its Customizable Keyword Localization System (CKWS) architecture. This may involve system investigations to attempt to improve keyword recognition accuracy by using short-term and long-term speaker features.

[0015] Therefore, in one aspect, a method for recognizing keywords in speech includes: detecting a first phoneme in the speech; converting the first phoneme into a first character; adding a second character from a dictionary to the first character to present an expanded character set; converting the expanded character set into a second phoneme set; pruning the second phoneme set to present an output phoneme; and recognizing keywords at least in part based on the output phoneme. The method can be performed using at least one neural network suitable for human speech generation, and the keywords in the speech can be indicated by the user and cannot be predetermined before being indicated in the speech.

[0016] In a non-restrictive example, the dictionary may include base words and translation words. Each base word may be associated with "K" phoneme maps, and each translation word may be associated with "N" phoneme maps. Each base word may also be associated with "M" translation words. "K", "N", and "M" may be integers, and at least "K" may be greater than 1.

[0017] In some implementations, the method may include instructing the user to say the keyword more than 1 predetermined number of times, and also instructing the user that the keyword should contain at least three phonemes.

[0018] Similarly, in some implementations, pruning the second phoneme set may include removing duplicate phoneme sequences from the second phoneme set. Pruning the second phoneme set may also include eliminating phoneme sequences from the second phoneme set if a particular phoneme sequence is shorter than a threshold sequence length. Furthermore, pruning the second phoneme set may include eliminating phoneme sequences from the second phoneme set if a first phoneme sequence in the second phoneme set is dissimilar to a second phoneme sequence constructed from the first phoneme. Similarity can be measured using fusion distance metrics based on Jaro Winkler normalized distance and Demerau Levenshtein normalized distance, such as equations. ,in are weighted parameters, where dl is the DemerauLevenshtein normalized distance and jw is the Jaro Winkler normalized distance.

[0019] Furthermore, in some examples, pruning the second phoneme set may include pruning based on sample matching of positive and negative samples, where phoneme sequences from the second phoneme set can be eliminated if they match only negative samples. If the weighted average of the matching samples is greater than a threshold, phoneme sequences from the second phoneme set that match both positive and negative samples may not be eliminated from the second phoneme set.

[0020] In some examples, a phoneme recognizer model can be used to perform the detection, and the method may further include training the phoneme recognizer model based on one or more records indicating a first phoneme after pruning. In these examples, the output phoneme can be a first output phoneme, the phoneme recognizer model can be a first phoneme recognizer model, and the training can render a second phoneme recognizer model from the first phoneme recognizer model. The method may then include: using the second phoneme recognizer model to render a second output phoneme after training, and then either adding the second output phoneme to the first output phoneme to attempt to recognize keywords based on both the first and second output phonemes, or replacing the first output phoneme with the second output phoneme to attempt to recognize keywords based on the second output phoneme instead of the first output phoneme.

[0021] In another aspect, a method for identifying keywords in speech includes: detecting a first phoneme sequence in the speech based on a keyword spoken in a first instance; using the first phoneme sequence to identify a first character sequence; and identifying at least a second and a third character sequence from a dictionary based on the first character sequence. The method further includes converting the second character sequence into at least a second and a third phoneme sequence and converting the third character sequence into at least a fourth and a fifth phoneme sequence. The method then further includes using at least one of the second, third, fourth, and fifth phoneme sequences to identify a keyword spoken again in a second instance, wherein the second instance occurs after the first instance.

[0022] Furthermore, in some example implementations, the method may include eliminating the second phoneme sequence and using at least one of the third, fourth, and fifth phoneme sequences but not the second phoneme sequence to identify the keyword spoken again in the second instance.

[0023] Furthermore, in some examples, a first phoneme recognizer model can be used to detect a first phoneme sequence, and the first phoneme sequence in the speech can be stored as one or more records. In these examples, the method may include using one or more records to adapt the first phoneme recognizer model to a person providing the speech. One or more records may be used to train the person's first phoneme recognizer model to present a second phoneme recognizer model, wherein the second phoneme recognizer model can be used to recognize keywords spoken in a second instance.

[0024] In another aspect, an apparatus includes at least one processor and at least one computer storage device that is not a transient signal. The computer storage device includes instructions executable by the at least one processor to register a wake word with a digital assistant using a first phoneme recognizer model based on a record of a person uttering the wake word, at least in part, by adding a first phoneme sequence derived from the record to a dictionary accessible to the first phoneme recognizer model. The instructions are also capable of training the first phoneme recognizer model using the record of the person uttering the wake word to present a second phoneme recognizer model. The instructions are then capable of replacing the first phoneme recognizer model with the second phoneme recognizer model, re-registering the wake word with the digital assistant based on the record but using the second phoneme recognizer model, and updating the dictionary by adding a second phoneme sequence derived from the record using the second phoneme recognizer model.

[0025] In some examples, the record can be the first record, the person can be the first person, and the wake word can be the first wake word. In these examples, the instructions can also be executed to register the second wake word with the digital assistant at least in part by adding a third phoneme sequence to the dictionary, based on the second record where the second person speaks the second wake word using a second phoneme recognizer model. The third phoneme sequence can be derived from the second record, where the second record may be different from the first record, the second person may be different from the first person, and the second wake word may be different from the first wake word. Also in these examples, the instructions can be executed to train the second phoneme recognizer model using the second record to present the third phoneme recognizer model, replace the second phoneme recognizer model with the third phoneme recognizer model, re-register the second wake word with the digital assistant based on the second record but using the third phoneme recognizer model, and update the dictionary by adding a fourth phoneme sequence derived from the second record using the third phoneme recognizer model.

[0026] The details of this application regarding both its structure and operation can be best understood by referring to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description

[0027] Figure 1 This is a block diagram of an example device network consistent with the principles of the present invention, wherein each device may include a keyword recognizer system as described herein;

[0028] Figure 2 and Figure 3 The architecture of a keyword location system according to the principles of the present invention is shown;

[0029] Figure 4 An example input / output phoneme and glyph pair for offline P2G model training according to the principles of the present invention is shown;

[0030] Figure 5 The illustration shows the actions taken during the first step of expanding the phonetic dictionary according to the principles of the present invention;

[0031] Figure 6 The illustration shows the actions taken during the second step of expanding the phonetic dictionary according to the principles of the present invention;

[0032] Figure 7 Examples of G2P and P2G variant files of various words / names according to the principles of the present invention are shown;

[0033] Figure 8 and Figure 9 An example flowchart is shown illustrating the actions taken during articulation augmentation and pruning based on P2G and G2P mapping according to the principles of the present invention.

[0034] Figure 10The illustration shows sample matching during articulation trimming according to the principles of the present invention;

[0035] Figure 11 An example flowchart of keyword adaptation for a phoneme recognizer model according to the principles of the present invention is shown;

[0036] Figure 12 An example architecture of a phoneme recognizer model according to the principles of the present invention is shown;

[0037] Figure 13A and Figure 13B A flowchart illustrating an example of the overall logic according to the principles of the present invention is shown;

[0038] Figure 14 An example flowchart illustrating the generation of phoneme variants of a word sequence using the sequitur tool according to the principles of the present invention is shown; and

[0039] Figure 15 This is an example overview flowchart of the instructions for decoding keywords according to the principles of the present invention. Detailed Implementation

[0040] This disclosure generally relates to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, distributed computer gaming networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural networks and machine learning applications. The systems described herein may include server and client components connected via a network to enable the exchange of data between the client and server components. The client components may include one or more computing devices, including game consoles (such as…) Or by or This includes game consoles manufactured by other manufacturers, virtual reality (VR) headsets, augmented reality (AR) headsets, game controllers, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers (such as laptops and tablets), and other mobile devices including smartphones and other examples discussed below. These client devices can operate in a variety of operating environments. For example, some client computers may employ, for example, Operating system, from Operating system, or Operating system, or by or The operating system produced. These operating environments can be used to execute one or more browsing programs, such as those generated by... or or The browser or other browser program that can access websites hosted by the internet servers discussed below. Furthermore, the operating environment based on this principle can be used to execute one or more computer game programs based on this principle.

[0041] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (such as...) Instantiation of (personal computers, etc.).

[0042] Information can be exchanged between clients and servers over a network. For this purpose and for security, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements a method for providing network users with a secure community for communication, such as an online social networking site or video game website, in accordance with these principles.

[0043] As used herein, instructions refer to computer-implemented steps for processing information within a system. Instructions can be implemented in software, firmware, or hardware and include any type of programming steps performed by system components.

[0044] The processor can be any conventional general-purpose single-chip or multi-chip processor, which can perform logic by means of various lines (such as address lines, data lines, and control lines) as well as registers and shift registers.

[0045] The software modules described in the flowcharts and user interfaces herein may include various subroutines, programs, etc. Without limiting this disclosure, logic stated to be executed by a particular module may be reassigned to other software modules and / or combined together in a single module and / or made available in a shareable library.

[0046] The principles of the invention described herein can be implemented as hardware, software, firmware, or a combination thereof; therefore, illustrative components, frames, modules, circuits, and steps are described in accordance with their functionality.

[0047] The functions and methods described below, when implemented in software, may be written in a suitable language such as, but not limited to, Java, C#, or C++, and may be stored on or transmitted through a computer-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disc read-only memory (CD-ROM) or other optical disc storage devices (such as Digital Universal Optical Disc (DVD)), magnetic disk storage devices, or other magnetic storage devices including removable thumb drives. Connections may be established on computer-readable media. Such connections may include, for example, hardwired cables, including fiber optic and coaxial cables, as well as digital subscriber line (DSL) and twisted-pair cables. Such connections may include wireless communication connections, including infrared and radio.

[0048] Components included in one embodiment may be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures may be combined, interchanged, or excluded from other embodiments.

[0049] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes the following systems: having only A; having only B; having only C; having both A and B; having both A and C; having both B and C; and / or having both A, B and C, etc.

[0050] Now for specific reference Figure 1An example system 10 is illustrated, which may include one or more of the example devices mentioned above and further described below according to the principles of the invention. The first example device included in system 10 is a consumer electronics (CE) device such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling the TV). However, the AVD 12 may alternatively be an appliance or household item, such as a computerized internet-enabled refrigerator, washer, or dryer. Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, augmented reality (AR) headset, virtual reality (VR) headset, internet-enabled or “smart” glasses, another type of wearable computerized device, such as, for example, a computerized internet-enabled watch, a computerized internet-enabled bracelet, other computerized internet-enabled devices, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that AVD 12 is configured to take the principles of the present invention (e.g., to communicate with other CE devices to take the principles of the present invention, to perform the logic described herein, and to perform any other functions and / or operations described herein).

[0051] Therefore, in order to implement this principle, AVD 12 can be... Figure 1Some or all of the components shown may be constructed. For example, AVD 12 may include one or more displays 14, which may be implemented as high-definition or ultra-high-definition (“4K”) or higher flat screens and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include: one or more speakers 16 for outputting audio according to these principles; and at least one additional input device 18 (such as, for example, an audio receiver / microphone) for, for example, inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, wide area network, local area network, etc.) under the control of one or more processors 24 (e.g., a central processing unit). Graphics processor 24A may also be included. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the principles of the invention, including other elements of AVD 12 as described herein, such as, for example, controlling display 14 to present images on the display and receive input from the display. Furthermore, it should be noted that network interface 20 may be, for example, a wired or wireless modem or router or other suitable interface (such as a wireless telephone transceiver, or a Wi-Fi transceiver as mentioned above).

[0052] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as a High Definition Multimedia Interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another CE device and / or a headphone port for connecting headphones to the AVD 12 to present audio from the AVD 12 to a user via headphones. For example, input ports 26 may be connected via wired or wireless connection to a wired or satellite source 26a of audio / video content. Thus, source 26a may be, for example, a standalone or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content such as computer game software and databases. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44, and may implement some or all of the logic described herein.

[0053] The AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. These computer memories 28 may, in some cases, be embodied as a stand-alone device within the AVD's housing, or as a personal video recording device (PVR) or video disk player for playing back AV programs, either inside or outside the AVD's housing, or as removable memory media. Furthermore, in some embodiments, the AVD 12 may include a location or position receiver (such as, but not limited to, a mobile phone receiver, a GPS receiver, and / or an altimeter 30), configured to receive geographic location information, for example, from at least one satellite or cell tower, and provide this information to the processor 24 and / or, in conjunction with the processor 24, determine the altitude at which the AVD 12 is set. However, it should be understood that, according to these principles, another suitable location receiver besides a mobile phone receiver, GPS receiver, and / or altimeter may be used, for example, to determine the location of the AVD 12 in all three dimensions.

[0054] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be, for example, thermal imaging cameras, digital cameras such as webcams, infrared (IR) cameras, and / or cameras integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or videos in accordance with these principles.

[0055] The AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element. Alternatively, it may use... .

[0056] Furthermore, the AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, gyroscopes, or magnetometers, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12.

[0057] Still referencing Figure 1In addition to AVD 12, system 10 may also include one or more other CE device types. In one example, a first CE device 44 may be used to send computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while a second CE device 46 may include components similar to the first CE device 44. In the illustrated example, the second CE device 46 may be configured as a VR headset worn by player 47 as shown, or a handheld game controller operated by player 47 with one or more joysticks, one or more buttons, etc. In the illustrated example, only two CE devices 44, 46 are shown; it should be understood that fewer or more devices may be used.

[0058] In the example shown, all three devices 12, 44, and 46 can be members of a recreational network, for example, in a home / residence or distributed across several locations. Therefore, unless otherwise expressly stated, this principle is not limited to the specific location indicated by dashed line 48.

[0059] The first CE device 44, as an example of a non-limiting device, can be constructed from any of the aforementioned devices, such as a portable wireless laptop computer or notebook computer, a video game console or video game controller, and therefore can have one or more components described below. The first CE device 44 can be a remote control (RC) for issuing commands such as AV play and pause to the AVD 12, or it can be a more complex device such as a tablet computer, a game controller communicating with the AVD 12 via a wired or wireless link, a personal computer, a wireless telephone, etc.

[0060] Therefore, the first CE device 44 may include one or more displays 50, which may be touch-enabled for receiving user input signals via touch on the displays 50. Alternatively, the displays 50 may be at least partially transparent displays configured to display AR and / or VR images, such as AR headset displays or "smart" glasses displays or "head-up" displays, and VR headset displays or other displays.

[0061] The first CE device 44 may further include one or more speakers 52 for outputting audio according to the principles, and at least one additional input device 54, such as an audio receiver / microphone, for inputting audible commands to the first CE device 44 to control the device 44, for example. The exemplary first CE device 44 may also include one or more network interfaces 56 for communicating over the network 22 under the control of one or more CE device processors 58. A graphics processor 58A may also be included. Thus, the interface 56 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including a mesh network interface. It should be understood that the processor 58 controls the first CE device 44 to implement the principles, including other elements of the first CE device 44 as described herein, such as, for example, controlling the display 50 to present images on the display and receive input from the display. Furthermore, it should be noted that the network interface 56 may be, for example, a wired or wireless modem or router, or other suitable interfaces, such as, for example, a wireless telephone transceiver, or a wireless network transceiver as described above.

[0062] In addition to the foregoing, the first CE device 44 may also include one or more input ports 60 (such as HDMI ports or USB ports, for example) for physically connecting (e.g., using a wired connection) to another CE device and / or for connecting headphones to the first CE device 44 to present audio from the first CE device 44 to a user via headphones. The first CE device 44 may also include one or more tangible computer-readable storage media 62, such as disk-based storage devices or solid-state storage devices. Furthermore, in some embodiments, the first CE device 44 may include a location or positioning receiver such as, but not limited to, a mobile phone and / or GPS receiver and / or altimeter 64, configured to receive geographic location information from at least one satellite and / or cell tower, for example, using triangulation, and provide said information to the CE device processor 58 and / or in conjunction with the CE device processor 58 to determine the altitude at which the first CE device 44 is set. However, it should be understood that, according to the principles of the invention, another suitable location receiver besides a mobile phone and / or GPS receiver and / or altimeter may be used to, for example, determine the location of the first CE device 44 in, for example, all three dimensions.

[0063] Continuing the description of the first CE device 44, in some embodiments, according to the principles of the invention, the first CE device 44 may include one or more cameras 66, which may be, for example, thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into the first CE device 44 and controllable by the CE device processor 58 to collect pictures / images and / or videos. The first CE device 44 may also include a Bluetooth transceiver 68 and other near-field communication (NFC) elements 70 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0064] In addition, the first CE device 44 may include one or more auxiliary sensors 72 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 may also include other sensors that provide input to the CE device processor 58, such as one or more climate sensors 74 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 76. In addition to the foregoing, it should be noted that in some embodiments, the first CE device 44 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 78, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the first CE device 44, and / or the CE device 44 may be powered via a wired connection to a wall socket. The CE device 44 may communicate with the AVD 12 via any of the communication modes and related components described above.

[0065] The second CE device 46 may include some or all of the components shown for CE device 44. One or both CE devices may be powered by one or more batteries and / or wall socket connections.

[0066] Referring now to the aforementioned at least one server 80, which includes at least one server processor 82, at least one tangible computer-readable storage medium 84 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 86, which, under the control of the server processor 82, allows communication on network 22 with... Figure 1 It can communicate with other devices, and in fact, this principle can facilitate communication between server and client devices. It should be noted that network interface 86 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0067] Therefore, in some implementations, server 80 may be an internet server or an entire server "farm" and may include and perform "cloud" functionality, enabling devices of system 10 to access the "cloud" environment via server 80, for example, in an example implementation of a network gaming application. Alternatively, server 80 may be configured with... Figure 1 Other devices shown are implemented in the same room or nearby using one or more game consoles or other computers.

[0068] In addition to the above, the logic blocks, modules, and circuits described below may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices (such as application-specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components, or any combination thereof) designed to perform the functions described herein. The processor may be implemented by a combination of a controller or state machine or computing device. Therefore, the methods described herein can be implemented as software instructions executed by a processor, a suitably configured application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) module, or any other convenient manner that will be understood by those skilled in the art. Where adopted, the software instructions may be embodied in a non-transitory device (such as a hard disk drive, CD-ROM, or flash drive). The software code instructions may also be downloaded from the Internet.

[0069] As understood in this paper, machine learning can be a serialized process involving identifying, acquiring, and preparing raw datasets; feature engineering and generating Extract, Transform, and Load (ETL) jobs that transform the raw data into features for machine learning models; and designing and encoding machine learning models. These models can be "trained" in computationally intensive tasks. The performance and latency of these models can even be evaluated.

[0070] Before proceeding with the description of the other figures, it should be understood from the principles of the invention that this application contemplates two phases. Specifically, this application contemplates both a registration phase / process and a detection phase / process. In the registration phase, the system may (e.g., audibly via a speaker or visually via a display) prompt the user to provide an audio sample of a keyword (e.g., specifically, a name) that they wish to use to trigger the assistant and respond by listening for subsequent commands from the user. The name provided by the user does not even need to be a pre-existing proper name, nor does it need to be an existing word in the user's language (or any language for that matter).

[0071] Furthermore, the system's prompts can instruct the user not only to pronounce the desired name for detection by the system's accessible microphones, but also to pronounce the desired name a number of times greater than a threshold of 1, such as five times. The prompts can also instruct the user to use keywords / names longer than a certain length, such as names with at least three syllables or letters (thus indicating at least three phonemes), because names with fewer than three phonemes may affect the accuracy of future name detections, potentially leading to false detections when the user may not have actually attempted to trigger the system.

[0072] In addition, when a user has a physical condition that may adversely affect the clear pronunciation of their name (such as a cold or sore throat), the prompt can also instruct the user not to register by saying their name a predetermined number of times.

[0073] Assuming the user has no physical condition affecting the pronunciation of their desired name, the user can pronounce the name a predetermined number of times as instructed by the system. The system can then record each pronunciation of the name based on speech detection via a microphone. The system can then register the name / keywords using the recorded speech waveforms to complete the registration phase without further user input.

[0074] Furthermore, it should be understood that during registration, simple consistency measurements can be used to eliminate outliers, such as the sounds of instances from which the user pronounces the expected name, which are significantly different from other instances of the user pronouncing the expected name that may also be recorded during the same registration period. Additionally, only pronunciations longer than a certain minimum phoneme length and consistent with other registered samples (other instances of the user pronouncing the name) are considered valid.

[0075] In short, the second stage (decoding stage) involves the subsequent real-time detection of the name spoken by the user. This may sometimes be referred to as "decoding" and can trigger the assistant to "listen" for subsequent commands or requests from the user to comply with. The second stage uses a phoneme recognizer model trained during the first stage and initially built using a sequence-to-sequence model (e.g., trained using connectionist temporal classification) and the TIMIT corpus (and / or several other corpora) to detect phonemes from the input speech. In the second stage, the detected phonemes can be used by a phoneme-level unique keyword decoder (e.g., a keyword matcher using phoneme sequence similarity measurements) to detect spoken keywords.

[0076] For reference Figure 2 and Figure 3 This illustrates the architecture of a keyword location system based on the principles of the present invention. Figure 2 It shows that in some respects, the form is more... Figure 3 A simpler architecture.

[0077] Firstly, from Figure 2Initially, it illustrates both the registration phase 200 and the detection / decoding phase 210. User 202 pronounces the desired name "Tom" three times 204, and then system 206 detects the name to present a speech waveform 208. As shown, in the registration phase, the name "Tom" pronounced by user 202 is input into a phoneme recognition AI model 212, which outputs a phoneme sequence indicating the phonemes recognized from the user's speech and provides the output to a pronunciation augmentation and trimming model 214 for augmenting and trimming the relevant phoneme sequence, which can be recognized as indicating the name "Tom" in the detection phase 210. Pronunciation augmentation and trimming will be discussed further below. Still refer to Figure 2 The pronunciation expansion and pruning model 214 can then be used to adjust the keyword dictionary 216 based on the expansion and pruning, wherein the base version of the dictionary 216 is pre-selected by the system developers and contains various character entries and one or more corresponding phonetic variants for each character entry.

[0078] Subsequently, keyword adaptation 218 can be performed, where a phoneme recognizer model 212 can be trained using warp data based on connectionist temporal classification (CTC) and user speech 204 as training data to adapt the phoneme recognizer model 212 to the user speech. The phoneme recognizer model 212 can then be replaced by an adapted phoneme recognizer model 220 for subsequent decoding of the name during the detection phase 210 when the name is spoken and used as streaming audio (however, in some implementations, the phoneme recognizer model 212 may also be used for decoding during the detection phase). Keyword adaptation may include modifying one or more phoneme recognizer models for the speaker's speech characteristics to improve keyword detection for that speaker. Therefore, the keyword-adapted model 220 can be used to improve the recognition accuracy of keyword detection, such as... Figure 2 and Figure 3 As shown, and in the detection phase 210, further adaptation can occur each time the system correctly identifies “Tom”, so that additional training can occur over time when the user triggers the assistant.

[0079] A more detailed description, such as Figure 2 As shown in detection phase 210, in this phase, user 202 can again speak the name provided to the system as a wake-up keyword, which can trigger the system to respond with an audio phrase 222, indicating that it has been triggered and is waiting for further user input for a task or command to be performed. Figure 2As shown in detection phase 210, the adapted phoneme recognizer 220 can be used to recognize the phrase “Tom” spoken by user 202 using keyword matcher / decoding module 224, so as to match the phonemes detected from the phrase “Tom” to entries in dictionary 216, thereby presenting decoded keywords 226 and triggering the system. The matching of keyword matcher / decoding module 224 can be performed by using streaming adaptive matching; and / or using a distance metric to evaluate the distance between the keyword phoneme sequence stored in dictionary 216 and the streaming phonemes output from adapted phoneme recognizer 220 for the phrase “Tom” spoken by user 202, to determine whether any entry in dictionary 216 will match the output within a threshold distance.

[0080] Figure 3 Specifically, it shows how to also register using intermittent keywords. However, in the detailed description... Figure 3 Previously, it should be understood that intermittent registration allows a user to register new keyword pronunciations at a later time. This can happen if he / she is dissatisfied with the wake-up response from a previously registered keyword, and if the system is "overly sensitive" and triggered by a false alarm when the user does not actually say the name they previously gave to the system, or if the system does not actually respond and wake up when the user says the keyword, in which case the user may become aware of this.

[0081] Users can reselect the registration mode for intermittent registration at any time they wish (e.g., by providing a voice command upon successful system triggering, based on input to a selector displayed on the screen to restart registration). Additionally or alternatively, the system itself can trigger intermittent registration if it determines that the user's voice has changed (e.g., due to no longer having a cold or sore throat as it might have when initial registration was performed). The system can also trigger intermittent registration if it determines that users have asked the same question more than one threshold number of times within each other's threshold timeframes, where the system identifies the question each time it is provided, even if the wake-up is triggered before or after the threshold number is reached. In any case, this new registration phase may occur similarly to the initial registration, which may require the user to speak the preferred keywords again within a valid threshold number of times.

[0082] Furthermore, it should be understood that during intermittent registration, users may even be allowed to provide different names or pronunciations as the system wake-up keyword, and after registering this different name, it can be used to trigger system wake-up, in addition to the previously provided name, which can still be used to trigger system wake-up. However, it should be further understood that during intermittent registration, users may provide additional samples of the same wake-up word name as provided during the initial registration.

[0083] Furthermore, in some implementations, different users can register differently (one of which may be intermittent), allowing different users to be assigned different names as corresponding wake words. These wake words can each trigger the same assistant / system, regardless of which user (or even another person) subsequently utters any registered name to trigger the system. Therefore, multiple different names / keywords can be assigned to the digital assistant to trigger it to take action based on verbal commands.

[0084] Similarly, during intermittent registration, users can be offered the option to retain previously registered instances (and append the newly registered pronunciation to the pronunciation dictionary) or to verify, check, and / or delete specific unwanted previously registered instances. This can be done through voice communication between the system and the user, via a graphical user interface (GUI) presented on a display, or similar means. For example, during registration verification in intermittent registration, a user can simply say his / her keywords a certain number of times, and the system may show him / her any matches against recognitions from previously registered records (e.g., waveform files). The user can then delete any previously registered instances that are returned as not matching the pronunciation provided at the start of registration verification.

[0085] Now, in detail... Figure 3 As shown, during intermittent registration 300, a user can speak a desired name / keyword a threshold number of times, using this as input 302. Then, a phoneme recognition model 304, which may have previously been or may not have been adapted to one or more users, can use this input to present a name specified as P. reg The pronunciation output 306 (e.g., “CHAH L IY”) can then be fed as input to the pronunciation augmentation and pruning model 308 to produce a pronunciation output P that may be greater than 1. aug The expansion set 310. Then, both the pronunciation output 306 and the expansion set 310 can be input into the dictionary 312.

[0086] Then, keyword adaptation 314 can occur, where phoneme recognizer model 304 can be retrained to present an adapted phoneme recognizer model 316, which can replace phoneme recognizer model 304 and be used in the detection phase 318. Also during adaptation 314, the same input 302 can be used again to present the pronunciation output, but instead of using model 316. This output can then be provided as input to pronunciation augmentation and pruning model 308 to present another augmented set of pronunciation output. These pronunciation outputs can then be used in the detection phase 318 to present a pronunciation P based on the output pronunciation P from the adapted phoneme recognizer model 316, given the input 320 from the user. test Determine the output pronunciation P testThe pronunciation similarity measurement 322 determines whether it is at least similar to a threshold amount T of these pronunciation outputs. As shown in the figure, if the similarity measurement is greater than or equal to the threshold T, the keyword 324 can be detected / recognized, or if the similarity measurement is less than the threshold T, the keyword 326 is not detected / recognized.

[0087] The following describes pronunciation expansion in more detail, particularly the expansion of the phonetic dictionary as an aspect of pronunciation expansion. See also: Figure 4 Furthermore, it should be reiterated that the system can be based on phoneme recognition. Therefore, a phoneme recognizer model can use the user's input speech to detect phoneme sequences (which may be building blocks of speech lexical representations). To enable the system to identify the presence of keywords in the user's speech, a phoneme-to-graph model can be used.

[0088] It should be understood that phoneme indication is very basic information, and the system can be configured to predict the many possible different pronunciations of a spoken keyword by training a neural network model using neural machine translation (NMT). This neural network model can learn to generate some or all possible combinations of character elements for use with speech input. In this paper, this is referred to as phoneme-to-character (P2G) model development.

[0089] To develop P2G models, one or more neural translation techniques can be used. For example... Figure 4 As the example shows, for a given word, there may exist various meaningful phonetic representations that represent the various possible ways the word can be pronounced. This itself may not always be a one-to-one mapping. However, for... Figure 4 Given a sequence of phonemes listed as input 400, the system can identify one or more corresponding approximate / nearest sequence of phonemes as output 402 to establish a one-to-one input / output pair, as shown in the figure.

[0090] Also Figure 4 As shown, these input / output pairs can be used to perform neural machine translation training 404 to develop a P2G model 406. The model may have been built by the system developers to take phoneme and epigraph pairs as input and target sequences. This developed phoneme-to-epithelial model can learn how to generate epigraph sequences for a given phoneme sequence. The model can be trained offline at the device containing the system without the help of a server.

[0091] from Figure 4Continuing with the principles of the present invention to help the reader understand, a P2G system can take a phoneme sequence and generate a translated sequence of glyphs based on that phoneme sequence. A G2P system can be considered the inverse of a P2G system and can use one or more systems from the text-to-speech domain. A G2P system can take a glyph sequence (or word, in some embodiments) and generate N possible phonetic variants for a given glyph sequence (or word). Such a G2P system can be used to generate, for example... Figure 4 The input-output pairs shown are used to construct the P2G model / system 406.

[0092] In addition, the phnseq used in this paper _phnrec It can be a phoneme sequence generated by a phoneme recognizer model through an input speech signal. The grapheme used in this paper _raw It is possible to establish a connection from phnseq _phnrec The exported original sequence of character elements (or words). phnseq _g2p It could be a phoneme sequence generated by a G2P system when a sequence of character elements is used as input. (grapheme) _trans It can be a sequence of morphemes translated from a phoneme sequence.

[0093] It should be further understood that if the grapheme sequence (grapheme _raw Phoneme sequences (phnseq) are obtained through the G2P system. _g2p ), and then the phoneme sequence is passed through the P2G system, which may yield a sequence that can be compared with the original grapheme sequence. _raw The sequence of graphemes for translation that matches or does not match. _trans This is because, as acknowledged in this application, such translations are not always absolute or perfect.

[0094] Considering the foregoing, now refer to Figure 5 In the first step, for example, the system developers can collect graphemes. _raw A relatively large list or dictionary 500 of words and / or proper nouns. For example, dictionary 500 could be built from 139,000 morphemes. This dictionary 500 can then be used by G2P system 502 to generate speech sequence variants (phnseq). _g2p )504, in the example implementation, can be limited to a maximum of five phonetic variants per character / word in dictionary 500. However, in other examples, a different threshold number of variants than five can be used, but in any case, the threshold number of variants can be used to reduce the number of variants to relax the processing constraints on the system and allow it to operate efficiently. Figure 5The text also shows example names / graphs 506 from the dictionary, as well as phonetic variants 508 for each name / graph 506.

[0095] Then, as Figure 6 The second step, as shown, involves generating the nearest translated character sequence / representation variant 602 for each phoneme sequence variant 504 obtained via the P2G system 600. Therefore, from... Figure 6 It can also be understood that the example speech sequence variant 604 can be mapped to one or more example translated character sequences / representation variants 606.

[0096] In the third step, a combined representation of steps one and two can be generated, such as... Figure 7 As shown. In Figure 7 In the example shown, the file can contain all G2P and P2G variants of all graphemes / words in a uniform representation. Therefore, the combined representation can include grapheme. _raw The first column of the entry is 700, Phnseq. _g2p The second column of the entry, 702, and grapheme _trans The third column of the entry is 704, where each row of entries maps to entries in other columns of the same row.

[0097] Subsequently, in the fourth step, assuming that one of the purposes of these steps might be to find the most suitable phonetic alternative to the spoken word during registration in order to improve detection accuracy, a P2G mapper development can be performed. Due to the potentially non-one-to-one mapping from the original grapheme to the translated grapheme, words / graphemes that sound similar but are different... _raw It may be translated into some of the same grapheme. _trans For example, a word like ["SHELLEY", "SHELLY"] might be translated as "SH ELLY". Therefore, one of the phoneme sequence variants of "Shelley" might be the same as one already translated as "Shelly". To centrally represent such phonetically similar words (or characters), a dictionary can be constructed in this fourth step, which returns a given translation grapheme. _trans All grapheme with similar pronunciations _raw Probability. For example, “SHELLY”: [“SHELLEY”, “SHELLY”], and “PORTA”: [“PUERTO”, “PORTA”, “PO RTO”].

[0098] Therefore, by following the steps described above and given a sequence of translated graphemes, it is possible to identify graphemes (or words) with similar pronunciations. _raw This allows us to identify all phonetic variants associated with those graphemes (or words). Therefore, it is possible to construct a system with all graphemes. _raw To speech variants and grapheme _trans To similar grapheme _raw A centralized dictionary. In some examples, this centralized dictionary may be referred to as a bidirectional P2G or G2P mapping dictionary.

[0099] While still describing pronunciation augmentation, but now specifically referring to the pronunciation augmentation using P2G and G2P mappings as disclosed above, it should be understood that during the registration phase, the system can receive five spoken pronunciations of keywords from the user (as described above). However, given that this may be a somewhat limited representation of the user-selected keywords relative to a typical speech-to-text model that may have been trained on hundreds or thousands of samples from many people using predefined keywords (e.g., predefined by the device manufacturer), problems such as repeatedly failing to recognize user keywords may arise if only these five samples from the user are used during the detection phase to identify keywords spoken by the user again. Furthermore, since user-selected keywords may be novel or unique, and thousands of samples may not be available as predefined by the developer or manufacturer, this application recognizes the advantage of generating many potentially “meaningful variations” of the user’s pronunciation of keywords to help the system better model and detect the user’s preferred keywords.

[0100] Therefore, refer to Figure 8 To help understand the expansion of speech variants at a "higher level". For example... Figure 8As shown, five registered pronunciations 800 from the user can be provided to the phoneme recognizer model 802 to output five corresponding phoneme sequences 804. These sequences 804 are then provided to the P2G translation model 806 so that the system can determine at diamond 808 whether any word translation from the P2G translation model 806 corresponds to a predetermined word / name / word in the keyword dictionary at least within a threshold similarity level such as 80 percent (e.g., if the actual keyword spoken by the user is unique, a fictional word, but still similar to an entry in the dictionary). If no word translation corresponds to a predetermined word / name / word in the keyword dictionary, then during the detection phase, only the speech sequences 804 of the five registered pronunciations are available for keyword localization. However, assuming that at least one word translation corresponds to a predetermined name or other word in the keyword dictionary, a bidirectional P2G, G2P mapping dictionary 810, as described herein, can be accessed to determine whether any alternative pronunciations exist at diamond 812, and in response to a positive determination at diamond 812, at step 814, the keyword dictionary can be expanded with alternative pronunciations to build a registered keyword speech dictionary 816. From there, duplicate speech sequences can be removed from the registered keyword speech dictionary 816 to present the final registered keyword speech dictionary 818 that can be used in the decoding stage.

[0101] Figure 9 This also illustrates the point. Phnseq of five audio samples from the user _phnrec 900 (in) Figure 9 P is specified in the middle. r This can be used as input to the P2G model 902 to output a grapheme. _trans Output 904 (in) Figure 9 W is designated as r Then you can use W. r The input is provided to the character expansion component / mechanism 906, which uses P2G and G2P dictionaries to represent grapheme-based... _trans Input 904 grapheme _raw Output 908. In some examples, component / mechanism 906 may consist of a neural network model, a text file, and / or an algorithm.

[0102] exist Figure 9 Lieutenant General grapheme _raw Output column 908 as W1 to W m Then, it is used as input to the speech sequence augmentation model 910 to generate speech sequences phnseq using P2G and G2P dictionaries. _g2p 912, including multiple corresponding speech sequences phnseq _g2p Corresponding to a grapheme_raw (For example, P) 11 -P 1n1 Corresponding to W1, and P 21 -P 2n2 (Corresponding to W2). Then, the speech sequence phnseq _g2p It can undergo a pruning process 914, which will be described further below, and finally present the final expanded speech sequence (P1-P). n )916, and then add it to the keyword speech dictionary.

[0103] Therefore, to achieve improved keyword coverage, a trained P2G model can be used to augment the registered sequences with more speech variants. It can now be understood that this process can be used to generate speech variants that represent the “most” suitable phoneme sequence of the registered keywords in the keyword / wake word dictionary used in the detection phase. It should be further understood that the principles of this invention can be extended to various languages ​​and platforms and help reduce computational complexity and improve detection accuracy.

[0104] The pronunciation pruning described below, as cited in this article, can be implemented in at least three types. Optionally, the three types described below can be implemented in the order presented below. But first, it should be understood that one reason for pronunciation augmentation is to cover reasonable differences in the same keyword, because during the detection phase, the same keyword may “sound” slightly different compared to the pronunciation at the time of registration, due to the speaker’s physical condition (e.g., sore throat), channel differences, or the user simply speaking the keyword slightly differently than during registration. It should be further understood that one reason for pronunciation pruning is to retain “good” augmented pronunciations to improve detection accuracy and remove “bad” augmented pronunciations to reduce false alarms / false activations by the assistant system.

[0105] Two of the three types of trimming mentioned above can be considered rule-based articulation trimming, and can be uniformly regarded as a "part" of trimming. In any case, the first type of trimming, belonging to rule-based articulation trimming, eliminates extended articulations shorter than the desired or minimum phoneme length. The second type of trimming, belonging to rule-based articulation trimming, eliminates extended articulations that derive from various extended articulations P. a The original decoded pronunciation P r Compared to expanded pronunciations that are not similar enough.

[0106] The first type of trimming only involves identifying augmented pronunciations that are shorter than the desired or minimum threshold phoneme length (e.g., set by the system developer). These may be eliminated and therefore not added to the keyword dictionary. Thus, this type of trimming may filter out augmented pronunciations that are too short.

[0107] The second type of pruning involves measuring P. r With Pa The similarity between them can be determined using fusion distance metrics based on Jaro Winkler normalized distance and Demerau Levenshtein normalized distance based on average. The fusion distance metric equation can be... ,in is a weighted parameter, where dl is the Demerau Levenshtein normalized distance and jw is the Jaro Winkler normalized distance. Therefore, with this type of pruning, if the resulting fusion distance is greater than a threshold distance (e.g., determined by the system developers), then augmented pronunciations that are too "far" (not similar enough) to the original decoded pronunciation may be filtered out.

[0108] The third type of trimming can be considered as sample-based pronunciation trimming. Here, pronunciation trimming can be based on using both positive and negative samples to go "deeper" than rule-based trimming. It involves preparing a set of positive data samples (based on the keywords themselves) and a set of negative data samples (non-keywords) to obtain sample match scores associated with matching pronunciations.

[0109] Positive samples can be built from keyword records (e.g., waveform files) recorded by the user during the first part of the registration phase and decoded using a phoneme recognition model, while negative samples can be selected from a subset of a word database collected offline and determined by the system developers. Optionally, these negative samples may sound significantly different from the keywords, making them not considered positive samples, but still not significantly different from the keywords, placing them near the keywords in the phonetic space and easily causing confusion. The negative data sample set can be extracted using phonetic distance measurements.

[0110] Sample matching can therefore be understood as obtaining the pronunciation P that is most similar to the decoded pronunciation of sample P0. a The process of *. Similarity / distance measurement can be Dist(P) a *, P0) = min (P a , P0), where P a It can be any extended pronunciation derived from P0.

[0111] The pronunciation trimming process based on positive / negative sample decoding is as follows: Figure 10As shown in the figure, if the expanded pronunciation 1000 (specified by a circle with a smaller radius) after rule-based pruning matches negative sample 1004 because it is most similar to the decoded pronunciation of one or more negative samples, then the pronunciation can be marked for deletion and ultimately removed. If the expanded pronunciation 1000 matches positive sample 1006 because it is most similar to the decoded pronunciation of one or more positive samples, then the pronunciation can be retained and ultimately added to the keyword dictionary. Furthermore, for clarity, it should be noted that the expanded pronunciation 1002 before rule-based pruning is specified by a circle with a larger radius, such as... Figure 10 As shown, the original decoded pronunciation P0 1008 is usually displayed in the center of these overlapping circles 1000, 1002.

[0112] However, it should be noted that in some scenarios, the same augmented pronunciation may match multiple samples, some of which may be positive and some of which may be negative. In this case, a weighted average score of the matching samples with the same augmented pronunciation can be used, and the augmented pronunciation can be preserved if the weighted average of the matching positive / negative samples is greater than a weighted average threshold.

[0113] For example, if the match is a negative sample, the weight might be negative, and if the match is a positive sample, the weight might be positive. However, when using a distance metric to determine distance, it's possible to intentionally ignore whether a sample weight is positive or negative. The distance can then be used to determine the magnitude of the weight; for example, the smaller the distance, the larger the weight, and vice versa. Therefore, it's understandable that if a match occurs with a negative sample that is close enough, the result might be a relatively large "negative" weight, but if the weighted average is still greater than a threshold, the amplified pronunciation might still be preserved.

[0114] Keyword adaptation will now be described, in which the system can be adapted to allow the user to provide the pronunciation of desired keywords during the registration phase, so that the system becomes more familiar with the user's voice. Specifically, keyword adaptation can be performed to train a general phoneme recognizer model (possibly initially used during the registration phase) using warp data based on connectionist temporal classification (CTC) and the user's newly registered keyword records as training data. This allows the phoneme recognizer model to be specifically adapted to the user's speech, further improving accurate keyword localization / decoding, and thus improving the user experience and interaction with the assistant / system.

[0115] Figure 11 The keyword adaptation process during the registration phase according to the principles of the present invention is illustrated. In a non-limiting embodiment, keyword adaptation may involve five steps. First, a “regular” registration 1100 from the input keyword record 1102 to the pronunciation dictionary 1104 can be performed as described above, just as if no keyword adaptation were performed. This may involve using a raw or generic phoneme recognizer model 1106 to represent the key specified as P.reg The pronunciation output 1107 (e.g., “CH AH L IY”) can then be fed as input to the pronunciation augmentation and pruning model 1108 to present a pronunciation output P that may be greater than 1. aug The extended set 1109. Both the pronunciation output 1107 and the extended set 1109 can be input into the dictionary 1104.

[0116] Secondly, keyword-pronunciation pairs can be used as training samples during the training / retraining phase 1110 to retrain / adapt the voicei recognizer model. Furthermore, it should be understood that after pronunciation augmentation and pruning 1108, some records may have multiple pronunciations associated with them; in this case, records can be copied / reused to create different record-pronunciation pairs for each pronunciation after augmentation and pruning.

[0117] Third, after training / retraining, the keyword-adapted phoneme recognizer model 1116 can replace the original phoneme recognizer model 1106, roughly as shown by arrow 1112.

[0118] Fourth, keyword pronunciations can be re-registered in the re-registration phase 1114, since the system may go through the same registration phases, including phoneme recognition 1118 and pronunciation expansion and phonology trimming 1120, but using a newer, keyword-adapted phoneme recognizer 1116.

[0119] Fifth, the keyword pronunciation dictionary 1104 can be updated as specified in box 1124, since the expanded previous pronunciation dictionary 1104 can be replaced with a new pronunciation dictionary 1122 generated based on step four in the preceding paragraphs. The previous pronunciation dictionary 1104 can then be deleted, or if the new pronunciation dictionary 1122 ultimately proves useless (e.g., if it is generated based on audio samples of a user when they have a sore throat or other illness affecting speech), it can be retained as a backup in system-accessible history / storage for later reuse. However, in other examples, the new dictionary 1122 (or at least new entries for dictionary 1122) can be appended to the previous keyword pronunciation dictionary 1104 that was expanded before keyword adaptation.

[0120] Furthermore, it should be understood that in some examples, where one or more users might assign multiple keywords / names to a single assistant / system, the above steps can be repeated for keyword matching. Therefore, keyword matching according to the above steps can be performed using records of one keyword assigned to the assistant (records from the same person and / or different people), and then the modified model can be modified again using records of a different keyword assigned to the assistant. For each set of records corresponding to a different keyword assigned to the assistant, and even for the same keyword, if the user intermittently registers as described above to provide a new set of records containing keywords previously provided by the user, matching can continue according to these steps. This helps the system identify anyone who utters any registered keywords for the assistant / system.

[0121] Figure 12 An example phoneme recognizer model architecture is illustrated, built from a recurrent neural network (RNN) that operates according to the principles of the present invention to receive a speech sequence 1200 as input and output one or more phoneme probabilities 1202 as output. As shown, the input 1200 can be passed to an input layer, then through multiple long short-term memory (“LSTM”) and batch normalization (“batch normalization”) hidden layers 1204, as well as one or more fully connected and calibrated linear unit (ReLU) layers 1206 and one or more fully connected and softmax function layers 1208, ultimately providing the output 1202 through an output layer.

[0122] from Figure 12 Continuing, it should be understood that another feature can also be used according to the principles of the invention. Specifically, during the decoding phase, pre-registered keyword localization can also be used for words that may accompany or surround a user-specified and potentially unique keyword. A separate dictionary of context-sensitive pre-registered keywords can be used for such purposes to further enhance the system's ability to distinguish between a false alarm where the user is actually attempting to trigger the system to wake up and a false alarm where the user does not intend to do so. Thus, for example, if the system identifies the unique keyword provided by the user with a certain probability, but that probability does not reach the probability threshold for waking up / activating the assistant solely based on a unique keyword, the system can consider the surrounding pre-registered keywords to determine whether they contain information requests or commands or other items that the system can perform, or rather, whether they only contain speech that is not intended to be input into the system. For example, "Tom, turn on the TV" could be an example of the former (where "Tom" is the wake-up keyword the user expects), while "Tom, my favorite color is green" could be an example of the latter. Thus, the first phrase is likely to trigger the system to wake up / activate, while the second phrase is unlikely to.

[0123] Now describing Figure 13A and Figure 13BThis diagram illustrates a flowchart of an example overall logic that can be executed by a digital assistant system based on the principles described herein. Starting at box 1300, the system can prompt the user to say a desired keyword / wake word a predetermined number of times, and can even prompt that the keyword should have at least a minimum length. At box 1302, the device can detect and record a first phoneme in the user's speech that indicates a non-predetermined keyword. Subsequently, at box 1304, the system can convert the first phoneme into a first glyph, and then at box 1306, use the first glyph to add a second glyph from a glyph dictionary to present an expanded glyph set.

[0124] The logic then proceeds from box 1306 to box 1308, where the system can convert the expanded character set into a second phoneme set. Subsequently, at box 1310, the system can trim the second phoneme set to present the output phonemes by eliminating, for example, repeating phoneme sequences, phoneme sequences shorter than a threshold, insufficiently similar phoneme sequences, and / or phoneme sequences that have already matched negative samples. Also at box 1310, the output phonemes can be added to the pronunciation dictionary.

[0125] The logic can continue from box 1310 to box 1312, where the system can use a warped record-based model (also from box 1302) trained with connectionist temporal classification (CTC) to train a phoneme recognizer model (used for phoneme detection in box 1302) to present the trained phoneme recognizer model, which can replace the initial phoneme recognizer model (used in box 1302) to re-register the user's keywords. The logic can then move to box 1314, where the system can add additional output phonemes to the pronunciation dictionary during re-registration based on the output from the trained phoneme recognizer model.

[0126] The logic can continue from box 1314 to box 1316, where, during the detection phase, the system can use the output from the trained phoneme recognizer model and a pronunciation dictionary with added pronunciations to identify keywords spoken by the user or another person. Then, if needed, the logic can move to box 1318, where the system can repeat steps 1300 to 1314, for example, during intermittent registration for different keywords or the same keywords.

[0127] It will be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements may be used to achieve the subject matter claimed herein.

Claims

1. An apparatus comprising: At least one processor; as well as At least one computer storage device, which is not a transient signal and includes instructions executable by the at least one processor to: Using a first phoneme recognizer model, based on a record of a person uttering a wake word, the wake word is registered to the digital assistant, at least in part, by adding a first phoneme sequence derived from the record to a dictionary accessible to the first phoneme recognizer model. The first phoneme recognizer model is trained using the record of the person saying the wake word to present the second phoneme recognizer model; Replace the first phoneme recognizer model with the second phoneme recognizer model; Based on the record, but using the second phoneme recognizer model, the wake word is re-registered with the digital assistant; as well as The dictionary is updated by adding a second phoneme sequence derived from the record using the second phoneme recognizer model.

2. The device of claim 1, wherein the record is a first record, wherein the person is a first person, wherein the wake word is a first wake word, and wherein the instruction is executable to: Using the second phoneme recognizer model, based on a second record of a second person speaking a second wake word, the second wake word is registered to the digital assistant at least in part by adding a third phoneme sequence derived from the second record, which is different from the first record, the second person is different from the first person, and the second wake word is different from the first wake word.

3. The device as claimed in claim 2, wherein the instructions are capable of executing to: The second phoneme recognizer model is trained using the second record to present the third phoneme recognizer model.

4. The device of claim 3, wherein the instructions are capable of executing to: Replace the second phoneme recognizer model with the third phoneme recognizer model.

5. The device of claim 4, wherein the instructions are capable of executing to: Based on the second record, but using the third phoneme recognizer model, the second wake word is re-registered with the digital assistant.

6. The device of claim 5, wherein the instructions are capable of executing to: The dictionary is updated by adding a fourth phoneme sequence derived from the second record using the third phoneme recognizer model.

7. A method comprising: Using a first phoneme recognizer model, based on a record of a person uttering a wake word, the wake word is registered to the digital assistant, at least in part, by adding a first phoneme sequence derived from the record to a dictionary accessible to the first phoneme recognizer model. The first phoneme recognizer model is trained using the record of the person saying the wake word to present the second phoneme recognizer model; Replace the first phoneme recognizer model with the second phoneme recognizer model; Based on the record, but using the second phoneme recognizer model, the wake word is re-registered with the digital assistant; as well as The dictionary is updated by adding a second phoneme sequence derived from the record using the second phoneme recognizer model.

8. The method of claim 7, wherein the record is a first record, the person is a first person, the wake word is a first wake word, and the instruction is executable to: Using the second phoneme recognizer model, based on a second record of a second person speaking a second wake word, the second wake word is registered to the digital assistant at least in part by adding a third phoneme sequence derived from the second record, which is different from the first record, the second person is different from the first person, and the second wake word is different from the first wake word.

9. The method of claim 8, further comprising: The second phoneme recognizer model is trained using the second record to present the third phoneme recognizer model.

10. The method of claim 9, further comprising: Replace the second phoneme recognizer model with the third phoneme recognizer model.

11. The method of claim 10, further comprising: Based on the second record, but using the third phoneme recognizer model, the second wake word is re-registered with the digital assistant.

12. The method of claim 11, further comprising: The dictionary is updated by adding a fourth phoneme sequence derived from the second record using the third phoneme recognizer model.

13. An apparatus comprising: At least one computer storage device, said at least one computer storage device is not a transient signal and includes instructions that can be executed by at least one processor to: Using a first phoneme recognizer model, based on a record of a person uttering a wake word, the wake word is registered to the digital assistant, at least in part, by adding a first phoneme sequence derived from the record to a dictionary accessible to the first phoneme recognizer model. The first phoneme recognizer model is trained using the record of the person saying the wake word to present the second phoneme recognizer model; Replace the first phoneme recognizer model with the second phoneme recognizer model; Based on the record, but using the second phoneme recognizer model, the wake word is re-registered with the digital assistant; as well as The dictionary is updated by adding a second phoneme sequence derived from the record using the second phoneme recognizer model.

14. The device of claim 13, wherein the record is a first record, wherein the person is a first person, wherein the wake word is a first wake word, and wherein the instruction is executable to: Using the second phoneme recognizer model, based on a second record of a second person speaking a second wake word, the second wake word is registered to the digital assistant at least in part by adding a third phoneme sequence derived from the second record, which is different from the first record, the second person is different from the first person, and the second wake word is different from the first wake word.

15. The device of claim 14, wherein the instructions are capable of executing to: The second phoneme recognizer model is trained using the second record to present the third phoneme recognizer model.

16. The device of claim 15, wherein the instructions are executable to: Replace the second phoneme recognizer model with the third phoneme recognizer model.

17. The device of claim 16, wherein the instructions are executable to: Based on the second record, but using the third phoneme recognizer model, the second wake word is re-registered with the digital assistant.

18. The device of claim 17, wherein the instructions are executable to: The dictionary is updated by adding a fourth phoneme sequence derived from the second record using the third phoneme recognizer model.

Images