Adaptive recognition of speech using key phrases

Abstract

The present disclosure relates to adaptively recognizing speech using key phrases. An example apparatus for recognizing speech includes an audio receiver to receive an audio stream. The apparatus also includes a key phrase detector to detect a key phrase in the audio stream. The apparatus also includes a model adapter to dynamically adapt a model based on the detected key phrase. The apparatus also includes a query recognizer to detect, by the adapted model, a voice query in the audio stream following the key phrase.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to the adaptive recognition of speech using key phrases. Background Technology

[0002] Speech recognizers are used to detect statements in audio. For example, statements may include commands that can be used to perform one or more tasks. Summary of the Invention

[0003] According to one aspect of this disclosure, an apparatus is provided. The apparatus includes: an audio receiver for receiving an audio stream; a key phrase detector for detecting key phrases in the audio stream; a model adapter for dynamically adapting a model based on the key phrases; and a query recognizer for detecting sound queries following the key phrases in the audio stream using the model.

[0004] According to another aspect of this disclosure, a method is provided. The method includes: receiving an audio stream via a processor; detecting key phrases in the audio stream via the processor; dynamically adapting a model based on the key phrases via the processor; and detecting sound queries following the key phrases in the audio stream via the model via the processor.

[0005] According to another aspect of this disclosure, a computer-readable storage device is provided. The storage device includes instructions that, when executed, cause one or more processors to detect at least a key statement in an audio stream; dynamically adapt a model based on the key statement; and detect sound queries in the audio stream following the key statement through the model.

[0006] According to another aspect of this disclosure, an apparatus is provided. The apparatus includes: means for receiving an audio stream; means for detecting key phrases in the audio stream; means for dynamically adapting a model based on the key phrases; and means for detecting sound queries following the key phrases in the audio stream using the model. Attached Figure Description

[0007] Figure 1 This is a block diagram illustrating an example system for spoken language understanding using adaptive speech recognition; Figure 2 This is a block diagram illustrating an example adaptive speech recognizer; Figure 3 This is a timing diagram illustrating an example process for adaptive speech recognition of spoken sentences; Figure 4 This is a block diagram illustrating an example adaptive speech recognizer that utilizes ultra-low power wake-on-voice. Figure 5 This is a timing diagram illustrating an example process for adaptive speech recognition using ultra-low power voice wake-up; Figure 6 This is a flowchart illustrating a method for adaptive speech recognition using key phrases; Figure 7 This is a block diagram illustrating an example computing device that can adaptively recognize speech using key phrases; and Figure 8 This is a block diagram illustrating a computer-readable medium storing code for adaptive key-state speech recognition.

[0008] Throughout this disclosure and the accompanying drawings, the same numerals are used to refer to similar components and features. The numerals in the 100 series refer to those originally used in [the original text / document]. Figure 1 Features found; the numbers in the 200 series refer to those originally found in Figure 2 The features found; and so on. Detailed Implementation

[0009] Speech recognizers can be trained on huge corpora of audio recordings from many people. Such speech recognizers can identify similar people speaking in similar acoustic environments. However, different acoustic environments or people with significantly different voices can lead to reduced recognition accuracy. For example, background noise can result in less accurate recognition.

[0010] This disclosure generally relates to techniques for adaptively recognizing speech using key phrases. For example, a key phrase may be a wake word detected by voice wake-up, or a pre-selected phrase that is easier to recognize. Specifically, the techniques described herein include apparatus, methods, and systems for adaptively recognizing speech using key phrases. An example apparatus includes an audio receiver for receiving an audio stream. The apparatus also includes a key phrase detector for detecting key phrases in the audio stream. The apparatus further includes a model adapter for dynamically adapting a model based on the detected key phrases. The apparatus also includes a query recognizer for detecting sound queries following the key phrases in the audio stream using the adapted model.

[0011] The techniques described in this paper use key phrases, such as wake words, to adapt the model to the speaker and the acoustic environment. The adapted model can then be used to more accurately detect commands following the key phrases. This adaptation can have a direct impact on the phrases following the wake word and any sound queries that follow it. Therefore, these techniques achieve a more reliable adaptation process because they do not assume that sufficiently good pseudo-real values ​​are generated in the first pass. Thus, the techniques described in this paper achieve improved speech recognition accuracy. Furthermore, these techniques can introduce minimal latency because they do not apply re-scoring. Instead, this technique achieves dynamic adaptation of the model used to detect sound queries by using backpropagation through a sequence of wake word subphone units.

[0012] Figure 1 This is a block diagram illustrating an example system for spoken language understanding using adaptive speech recognition. It can be used... Figure 6 Method 600 in Figure 7 Example system 100 is implemented in computing device 700.

[0013] Example system 100 includes speech 102 being captured by audio capture component 104. For example, audio capture component 104 may be a microphone or microphone array. System 100 also includes an adaptive speech recognizer 106 communicatively coupled to audio capture component 104. For example, adaptive speech recognizer 106 may be a reference... Figure 2 and Figure 4 The adaptive speech recognizer 200 or 400 is described. System 100 includes an application 108 communicatively coupled to the adaptive speech recognizer 106.

[0014] exist Figure 1 In example system 100, audio capture component 104 records audio corresponding to speech 102 and provides a speech signal with a suitable format. For example, this format may include speech features, digital audio, or any other suitable format. The speech signal is received by wake-word adaptive speech recognizer 106. Wake-word adaptive speech recognizer 106 processes the speech signal and recognizes the spoken words. The recognized words are then sent to application 108. For example, application 108 may be a natural language understanding application that can determine intent based on one or more detected commands from the recognized words. In various examples, application 108 may be a virtual assistant application.

[0015] In various examples, the voice query in speech 102 can follow the template: <wake-up statement> <voice command>. For example, "<wake-up statement>" := "Hello Computer" and "<voice command>" := "Where is the next restaurant?" The voice query can be captured by audio capture device 104 and sent to wake-up word adaptive speech recognizer 106. Wake-up word adaptive speech recognizer 106 can detect the wake-up statement "Hello Computer" and use the detected wake-up statement to improve the recognition accuracy of the voice command "Where is the next restaurant?" that follows. In various examples, wake-up word adaptive speech recognizer 106 adapts the speech recognition model to the speaker and acoustic environment based on the detected wake-up word. Wake-up word adaptive speech recognizer 106 can calculate the adaptation direction by analyzing the wake-up triggering process.

[0016] The wake-word adaptive speech recognizer 106 can be configured in various ways. For example, in a combined low-power configuration, wake-word detection and voice query recognition can share the same neural network. However, the decoding may differ, resulting in lower power consumption for wake-word detection. For instance, the decoder may only consider senones or phonemes present in key phrases or wake-words. ASR needs to consider any phonemes in the vocabulary. Therefore, wake-word detection can be performed with much lower computational and memory usage. (See reference...) Figure 2 and Figure 3 An example wake-word adaptive speech recognizer 106 configured with a combination of low-power settings is described.

[0017] In some examples, the wake-word adaptive speech recognizer 106 can be configured in a dedicated ultra-low power configuration. For example, wake-word detection can be performed at a dedicated component. As an example, this dedicated component can operate in ultra-low power mode. See reference. Figure 4 and Figure 5 An example wake-word adaptive speech recognizer 106 configured with a dedicated ultra-low power configuration is described.

[0018] Figure 1 The diagram is not intended to indicate that the example system 100 includes Figure 1 All the components shown. Instead, fewer components or... Figure 1 Example system 100 is implemented using additional components not shown (e.g., additional speech, capture devices, adaptive speech recognizers, applications, etc.).

[0019] Figure 2 This is a block diagram illustrating an example adaptive speech recognizer. It can be used... Figure 6 Method 600 in Figure 7 An example adaptive speech recognizer 200 is implemented in the computing device 700.

[0020] Example adaptive speech recognizer 200 includes a feature front-end 202. Adaptive speech recognizer 200 includes a model R 204 communicatively coupled to the feature front-end 202. In various examples, model R 204 may be a neural network. For example, model R 204 may be a recurrent neural network, a quasi-recurrent neural network, a convolutional neural network (CNN), or a time-delayed neural network (TDNN). Model R 204 includes an acoustic model 206 and a language model 208. For example, acoustic model 206 is the state of audio features given a sequence of phonemes or words represented by a sequence of phonemes. In various examples, language model 208 may compute the probability that a recognized sequence of words may or may not appear in the context. For example, model R 204 may generate multiple hypotheses, and language model 208 may compute the probability of the correctness of each hypothesis. In some examples, a decoding process is also included. For example, the decoding process may include Viterbi beam search or greedy search. In some examples, language model 208 can be a weighted finite-state transducer or a neural network. In various examples, the output of acoustic model 206 can be multiplied by the output of language model 208 to generate the most probable word sequence.

[0021] exist Figure 2 In the example, the feature front-end 202 receives the speech signal and computes the speech feature stream. For example, the speech feature stream could be multiple mel-filter banks. The speech feature stream is then received by model R 204. As an example, model R 204 could be a recurrent neural network R. Model R 204 can have two inputs, "h" and "f", at each time step. Given the history vector "h" and the feature vector "f", and the initial state h0 of model R 204, acoustic model 206 computes the output for each time step, as shown in the reference... Figure 3As described in more detail. In some examples, the history vector h is the recurrence signal passed from one time step to another in a recurrent neural network (RNN). Therefore, the history vector h can depend on the actual implementation of the RNN in various examples. In some examples, when using a Long Short-Term Memory (LSTM) RNN or a Gated Recurrent Unit (GRU) RNN, the history vector can be the result of an affine transformation with or without activation against the Elman-RNN, or a concatenation of multiple vectors from multiple affine transformations with or without activation. For example, each output can be a probability distribution over multiple phonemes. As used in this paper, a multiple phoneme refers to an acoustic entity that is more granular than a phoneme. The multiple phonemes can be matched to speech segments using an acoustic model 206. For example, for each speech segment (frame), the acoustic model 206 can be used to compute a probability distribution over multiple phonemes. The model R 204 can then use this distribution to compute the final letter sequence. In some examples, the model R 204 may also include a weighted finite-state transducer used in conjunction with beam search. In various instances, the recurrent neural network of model R 204 can be used to compute the final letter sequence given a language model 208.

[0022] In some examples, model R 204 can be a TDNN. For example, a TDNN can operate with multiple interconnected perceptron layers. A TDNN can also be implemented as a feedforward neural network. In various examples, all neurons at each layer of the TDNN receive input from the output of neurons in the layer below. All units at each layer of the TDNN receive input from a context window of the output from the layer below. For time-varying signals (e.g., speech), each unit has a connection to the output from the unit below it, and also has connections to the time-delayed past outputs of these same units. This models the temporal pattern or trajectory of the units. Furthermore, translation invariance can be achieved by explicitly removing positional dependencies during the backpropagation training process. Removing positional dependencies is done by making a time-shifted copy of the network in the invariant dimension (e.g., time). The error gradient is then computed from the overall target vector using backpropagation through all these networks. The error gradients associated with the translated copies are averaged before weight updates are performed, and therefore they are shared and constrained to be equal. Therefore, all positional dependencies from backpropagation training via translated copies are removed, and the copied network learns the most salient hidden features without translation invariance. For example, the most salient hidden features may be independent of their exact location in the input data.

[0023] Figure 2 The diagram is not intended to indicate the example adaptive speech recognizer 200 including Figure 2All the components shown. Instead, fewer components or... Figure 2 Additional components (e.g., additional feature front-ends, models, etc.) not shown are used to implement the example adaptive speech recognizer 200. In some examples, the adaptive speech recognizer 200 may also include a natural language understanding unit.

[0024] Figure 3 This is a timing diagram illustrating an example process for adaptive speech recognition using key phrases. It can be used... Figure 2 Example Adaptive Speech Recognition 200 Figure 7 Computing device 700, or Figure 8 Computer-readable media 800 Figure 1 Example process 300 is implemented in system 100.

[0025] Figure 3 Includes voice 302, which includes a wake-up statement 304 and a command 306. Figure 3 In the example, wake-up statement 304 is "Hello computer" and command 306 is "Turn on the light". Figure 3 Includes historical vectors 308A, 308B, 308C, 308D, 308E, and 308F, which represent the state of model R at times 310A, 310B, 310C, 310D, 310E, and 310F. Figure 3 It also includes feature vectors 312A, 312B, 312C, 312D, 312E, and 312F received at model R at times 310A, 310B, 310C, 310D, 310E, and 310F, respectively.

[0026] At arrow 314, a forward pass, including key statement detection, is performed regardless of the speaker. This occurs at time steps t0 to t... n The system identifies the wake-up phrase "Hello Computer". This forward pass can be a neural network inference step. In various examples, this forward pass can be performed on a low-power platform. For example, the low-power platform can be a digital signal processor (DSP).

[0027] At arrow 316, the detected key statement is sent to initialize the speaker-related model. In response to t n A critical statement was detected at the location, and process 300 can initiate a backward propagation.

[0028] At arrow 318, a backpropagation is performed. For example, after a key statement is detected, this observation is used to initialize the speaker-related model by adjusting the initial state h0. For example, in the backpropagation, the error at each time step is backpropagated to the initial state h0. In various examples, the error can be calculated by comparing the probability distribution of the identified multi-phonemes of the key statement spoken by a particular speaker to the optimal distribution. Therefore, the initial state h0 can be updated to minimize the error observed by identifying a particular speaker. In some examples, in addition to adjusting the initial state h0, or as an alternative, process 300 may include updating all weights in model R. In various examples, the trade-off between adjusting all weights in model R or only a subset of weights can be configured based on the target platform. As an example, this subset could be only the weights of the initial state h0.

[0029] At arrow 322, a speaker-related fast forward pass is performed. For example, the entire voice query is identified in a speaker-related manner. In some examples, the speaker-related forward pass 322 can correct for false triggering of the speaker-independent forward pass 314 to improve overall accuracy.

[0030] Figure 3 Graph 324 shows the expected CPU load at different times in process 300. From time t0 to time t... n At time t, in ultra-low power state 326, the CPU load is minimal. n At time t, the CPU load peaks at high power (328), during which time losses are calculated and the initial state is updated to h0. n Then at time t N During the period between these times, the system is in a low-power state 330 with low CPU load, and command 306 is detected in low-power state 330. In various examples, the expected CPU load and associated power consumption may differ for different platform configurations. For example, in a given DSP mode, the expected power consumption may depend on the memory settings.

[0031] This process flowchart is not intended to indicate that the blocks of example process 300 are executed in any particular order, or to include all blocks in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included within example process 300.

[0032] Figure 4 This is a block diagram illustrating an example adaptive speech recognizer that utilizes ultra-low power sound wake-up. It can be used... Figure 6 Method 600 in Figure 7 Example system 400 is implemented in computing device 700.

[0033] Example adaptive speech recognizer 400 includes from Figure 2 Similar numbered components. Furthermore, the adaptive speech recognizer 400 includes an ultra-low power wake-up (WOV) 402. For example, the ultra-low power wake-up (WOV) 402 can operate while the rest of the system is powered off. Therefore, during the operation of the WOV 402, only the WOV 402 may consume battery power. When the WOV 402 component detects a key phrase, as described below, the system 400 can move to a low-power consumption state to re-evaluate the audio for use in the complete recognition / understanding process, including adaptation. In some examples, the ultra-low power WOV 402 uses a very small neural network that is highly optimized for a specific task. In various examples, it may also be optimized for specific hardware. For example, the task may be optimized to utilize hardware acceleration of the neural network for processing.

[0034] exist Figure 4 In the example, the Low Power Wake-up Voice (WOV) 402 can detect a wake-up statement in the audio stream. The detected wake-up statement from the WOV 402 can then be used to adapt model R 204. Therefore, in this example, model R 204 does not detect a wake-up statement and begins execution in response to receiving a wake-up statement from the WOV 402.

[0035] In various examples, the adaptive speech recognizer 400 can be used Figure 5 The process involves adapting the R204 model to 500 and performing speech recognition.

[0036] Figure 4 The diagram is not intended to indicate the example adaptive speech recognizer 400, which includes Figure 4 All the components shown. Instead, fewer components or... Figure 4 Additional components (e.g., additional feature front-ends, models, etc.) not shown are used to implement the example adaptive speech recognizer 400. In some examples, the adaptive speech recognizer 400 may also include a natural language understanding unit.

[0037] Figure 5 This is a timing diagram illustrating an example process for adaptive speech recognition using ultra-low power sound wake-up. It can be used... Figure 4 An example of an adaptive speech recognizer 400 that utilizes ultra-low power sound wake-up. Figure 7 Computing device 700, or Figure 8 Computer-readable media 800 Figure 1 Example process 500 is implemented in system 100.

[0038] Figure 5 The process 500 includes Figure 3 Similar numbered components. However, in process 500, model R can be at time t. n After receiving the wake-up message "Hello Computer" detected in the example, execution begins.

[0039] exist Figure 5 In example process 500, a complete speaker-independent wake-up detection process is performed in the ultra-low power model. Triggered wake-up word detection 304 prompts forward / backward propagation to determine the speaker-dependent model R. Then, a speaker-dependent voice query 306 is identified.

[0040] At arrow 502, a forward pass is performed. Figure 5 In the example, the forward pass includes the pass to feature f n Backward decoding to f0. For example, the loss can be calculated for the detected wake-up statement "Hello Computer" 304. At arrow 504, the calculated loss is sent for backward transmission.

[0041] At arrow 506, a backpropagation is performed. For example, a backpropagation involves updating the state h based on the calculated loss. n+1 Therefore, the objective of process 500 could be to adjust state h. n+1 Make adjustments, rather than as in Figure 3 The update h0 is described in process 300. At arrow 508, the updated state h is sent. n+1 , for use in rapid forward transmission.

[0042] At arrow 510, a fast forward pass is performed. For example, based on the updated state h. n+1 Command “Turn on the light” 306 detected. This process flowchart is not intended to indicate that the blocks of example process 500 are executed in any particular order, or to include all blocks in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included in example process 500.

[0043] Figure 5 Graph 512 shows the expected CPU load at different times in process 500. From time t0 to time t... n At this point, under ultra-low power condition 326, the CPU load is minimal. At time t... n At point 514, the CPU load peaks in high-power state 514, during which time losses are calculated and the initial state is updated to h0. However, the CPU load in high-power state 514 can be lower than... Figure 3 CPU load at high power level 328. At time t... n Then at time t NDuring the time interval, in low-power state 330, the CPU load is low, and command 306 is detected in low-power state 330. Similarly, in various examples, the expected CPU load and associated power consumption may differ for different platform configurations.

[0044] Figure 6 This is a flowchart illustrating a method for adaptively recognizing speech using key phrases. It can be used... Figure 2 and Figure 4 Adaptive speech recognizer 200 or adaptive speech recognizer 400, Figure 7 Computing device 700, or Figure 8 Computer-readable media 800 Figure 1 Example method 600 is implemented in system 100.

[0045] At block 602, an audio stream is received. For example, the audio stream could be captured speech.

[0046] At block 604, a key statement is detected in the audio stream. For example, a forward pass can be performed on the model to detect the key statement. In various examples, the key statement can be a wake-up statement. For instance, the audio stream can be processed using an ultra-low power wake-up statement detector to detect the key statement.

[0047] At block 606, the model is dynamically adapted based on the detected key phrases. In some examples, the initial state of the model is adjusted during backpropagation. For example, at each time step, the error can be backpropagated back to the initial state. In some examples, multiple weights of the model can be adjusted during backpropagation. In various examples, the error can be computed by comparing the identified multi-phoneme probability distribution of key phrases spoken by a particular speaker with the optimal distribution. For example, the multi-phoneme probability distribution can be more granular than phonemes or word sequences. In various examples, any level of granularity can be used. In some examples, the error can be computed in parallel and at different granularity levels.

[0048] At block 608, an adapted model is used to detect sound queries following key statements in the audio stream. In some examples, sound queries can be detected by generating probability distributions over multi-phonemes. In various examples, sound queries can be detected by computing the final letter sequence.

[0049] This flowchart is not intended to indicate that the blocks of example method 600 should be executed in any particular order, or that all blocks should be included in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included within exemplary method 600. For example, method 600 may include generating a speech feature stream based on an audio stream. Key phrases are then detected based on the feature stream.

[0050] For reference Figure 7 , Figure 7 This is a block diagram illustrating an example computing device 700 that can adaptively recognize speech using key phrases. The computing device 700 can be, for example, a laptop computer, desktop computer, tablet computer, smart TV, mobile device, or wearable device, etc. In some examples, the computing device 700 can be a virtual assistant application. In various examples, the computing device 700 can be a medical device. The computing device 700 can include a central processing unit (CPU) 702 configured to execute stored instructions, and a memory device 704 storing instructions executable by the CPU 702. The CPU 702 can be coupled to the memory device 704 via a bus 706. Furthermore, the CPU 702 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Additionally, the computing device 700 can include more than one CPU 702. In some examples, the CPU 702 can be a system-on-a-chip (SoC) with a multi-core processor architecture. In some examples, the CPU 702 can be a dedicated digital signal processor (DSP) for image processing. Memory device 704 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 704 may include dynamic random access memory (DRAM).

[0051] Memory device 704 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 704 may include dynamic random access memory (DRAM).

[0052] The computing device 700 may also include a graphics processing unit (GPU) 708. As shown, the CPU 702 can be coupled to the GPU 708 via a bus 706. The GPU 708 can be configured to perform any number of graphics operations within the computing device 700. For example, the GPU 708 can be configured to render or manipulate graphical images, graphics frames, videos, etc., to be displayed to a user of the computing device 700.

[0053] Memory device 704 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 704 may include dynamic random access memory (DRAM). Memory device 704 may include device drivers 710 configured to execute instructions for training multiple convolutional neural networks to perform sequence-independent processing. Device drivers 710 may be software, application programs, application code, etc.

[0054] CPU 702 can also be connected via bus 706 to input / output (I / O) device interface 712, which is configured to connect computing device 700 to one or more I / O devices 714. I / O devices 714 may include, for example, a keyboard and pointing devices, wherein the pointing devices may include a touchpad or a touchscreen, etc. I / O devices 714 may be built into computing device 700 or may be devices connected externally to computing device 700. In some examples, memory 704 may be communicatively coupled to I / O devices 714 via direct memory access (DMA).

[0055] CPU 702 can also be linked to display interface 716 via bus 706, which is configured to connect computing device 700 to display device 718. Display device 718 may include a display screen, which is a built-in component of computing device 700. Display device 718 may also include a computer monitor, television, or projector, etc., connected to computing device 700 internally or externally.

[0056] The computing device 700 also includes a storage device 720. The storage device 720 is a physical memory, such as a hard disk drive, optical disk drive, thumb drive, drive array, solid-state drive, or any combination thereof. The storage device 720 may also include a remote storage drive.

[0057] The computing device 700 may also include a network interface controller (NIC) 722. The NIC 722 can be configured to connect the computing device 700 to a network 724 via a bus 706. The network 724 can be a wide area network (WAN), a local area network (LAN), or the Internet, etc. In some examples, the device can communicate with other devices wirelessly. For example, the device can communicate with other devices via a wireless LAN connection. In some examples, the device can connect to and communicate with other devices via Bluetooth® or similar technologies.

[0058] The computing device 700 also includes a microphone 726. For example, the microphone 726 may include one or more sensors. In some examples, the microphone 726 may include a processor for generating filtered audio. For example, noise cancellation may be applied to the filtered audio. In some examples, the microphone 726 may include beamforming to generate the filtered audio.

[0059] The computing device 700 also includes an adaptive speech recognizer 728. For example, the adaptive speech recognizer 728 can be used to adaptively recognize speech in an audio stream. The adaptive speech recognizer 728 may include an audio receiver 730, a feature front-end 732, a key phrase detector 734, a model adapter 736, and a query recognizer 738. In some examples, each component 730-738 of the adaptive speech recognizer 728 may be a microcontroller, an embedded processor, or a software module. The audio receiver 730 may receive an audio stream. The feature front-end 732 may compute a speech feature stream based on the received audio stream. For example, the speech feature stream may be multiple Mel filter banks. The key phrase detector 734 may detect key phrases in the audio stream. For example, the key phrase detector 734 may detect key phrases in the audio stream based on the speech feature stream. In some examples, the key phrase detector 734 may be a dedicated ultra-low power voice wake-up unit for detecting key phrases based on the speech feature stream. For example, the key phrase may be a wake-up statement. The model adapter 736 may dynamically adapt a model based on the detected key phrases. In some examples, the model includes an acoustic model for generating probability distributions over multiple phonemes. In various examples, the model includes a language model for computing the final letter sequence. In some examples, the model may be a recurrent neural network. In various examples, the model may be a time-delayed neural network. The query recognizer 738 can detect sound queries following key phrases in an audio stream using an adapted model.

[0060] The storage device 720 of the computing device 700 includes an application 740. For example, the application 740 can use a detected voice query to perform one or more actions. As an example, the application 740 could be a natural language understanding application. In various examples, the application 740 could be a virtual assistant application. In some examples, the application 740 could be any voice control application.

[0061] Figure 7 The block diagram is not intended to indicate that the computing device 700 includes Figure 7 All the components shown. Instead, computing device 700 may include fewer or... Figure 7 Additional components, such as additional buffers, additional processors, etc., are not shown in the diagram. In some examples, computing device 700 may include a digital signal processor for detecting key statements. In various examples, computing device 700 may include a neural network for detecting key statements and adapting the model. Depending on the details of the specific implementation, computing device 700 may include any number of... Figure 7Additional components are not shown. Furthermore, any functionality of the audio receiver 730, feature front-end 732, key phrase detector 734, model adapter 736, and query recognizer 738 can be implemented partially or entirely in hardware and / or in processor 702. For example, functionality can be implemented using an application-specific integrated circuit (ASIC), in logic implemented in processor 702, or in any other device. Similarly, any functionality of CPU 702 can be implemented partially or entirely in hardware and / or in processor. For example, the functionality of adaptive speech recognizer 728 can be implemented using an ASIC, in logic implemented in processor, in logic implemented in a dedicated graphics processing unit such as GPU 708, or in any other device.

[0062] Figure 8 This is a block diagram illustrating a computer-readable medium 800 storing code for adaptive key-state speech recognition. The computer-readable medium 800 can be accessed by a processor 802 via a computer bus 804. Furthermore, the computer-readable medium 800 may include code configured to instruct the processor 802 to execute the methods described herein. In some embodiments, the computer-readable medium 800 may be a non-transitory computer-readable medium. In some examples, the computer-readable medium 800 may be a storage medium.

[0063] like Figure 8As shown, the various software components discussed herein can be stored on one or more computer-readable media 800. For example, an audio receiver module 806 can be configured to receive an audio stream. A feature front-end module 808 can be configured to generate a speech feature stream based on the audio stream. A key phrase module 810 can be configured to detect key phrases in the audio stream. In some examples, the key phrase module 810 can be configured to detect key phrases based on the feature stream. In various examples, the key phrase module 810 can be configured to perform a forward pass on the model to detect key phrases. In some examples, the key phrase module 810 can be configured to process the audio stream using an ultra-low power wake-up phrase detector. In various examples, the key phrase module 810 can be configured to generate a speech feature stream based on the audio stream. For example, the key phrase module 810 can be configured to detect key phrases based on the feature stream. A model adapter module 812 can be configured to dynamically adapt the model based on the detected key phrases. For example, the model adapter module 812 may include code for propagating the error at each time step back to the initial state. In some examples, model adapter module 812 may include code for adjusting multiple weights of the model during backpropagation. In various examples, model adapter module 812 may include code for calculating error by comparing the identified multi-phoneme probability distributions of key phrases spoken by a particular speaker with an optimal distribution. In some examples, model adapter module 812 may include code for adjusting the initial state of the model during backpropagation. Query recognizer module 814 may be configured to detect sound queries following key phrases in an audio stream using an adapted model. For example, query recognizer module 814 may be configured to generate probability distributions over multi-phonemes. In some examples, query recognizer module 814 may be configured to detect sound queries by calculating a final letter sequence.

[0064] Figure 8 The block diagram is not intended to indicate that computer-readable media 800 includes Figure 8 All components are shown. Furthermore, depending on the details of the specific implementation, the computer-readable medium 800 may include any number of... Figure 8 Additional components not shown.

[0065] Example Example 1 is an apparatus for speech recognition. The apparatus includes an audio receiver for receiving an audio stream. The apparatus also includes a key phrase detector for detecting key phrases in the audio stream. The apparatus further includes a model adapter for dynamically adapting a model based on the detected key phrases. The apparatus also includes a query recognizer for detecting sound queries in the audio stream that follow the key phrases using the adapted model.

[0066] Example 2 includes the apparatus as described in Example 1, including or excluding optional features. In this example, the apparatus includes a digital signal processor for detecting key statements.

[0067] Example 3 includes an apparatus as described in any one of Examples 1 to 2, including or excluding optional features. In this example, the apparatus includes a neural network for detecting key statements and adapting the model.

[0068] Example 4 includes an apparatus as described in any one of Examples 1 to 3, including or excluding optional features. In this example, the apparatus includes a dedicated ultra-low power voice wake-up unit for detecting key statements based on a speech feature stream.

[0069] Example 5 includes a device as described in any one of Examples 1 to 4, including or excluding optional features. In this example, the key statement includes a wake-up statement.

[0070] Example 6 includes an apparatus as described in any one of Examples 1 to 5, including or excluding optional features. In this example, the apparatus includes a feature front-end for calculating a speech feature stream based on a received audio stream.

[0071] Example 7 includes an apparatus as described in any one of Examples 1 to 6, including or excluding optional features. In this example, the model includes an acoustic model for generating probability distributions over multiple phonemes.

[0072] Example 8 includes the apparatus as described in any one of Examples 1 to 7, including or excluding optional features. In this example, the model includes a language model for computing the final letter sequence.

[0073] Example 9 includes a device as described in any one of Examples 1 to 8, including or excluding optional features. In this example, the model includes a recurrent neural network.

[0074] Example 10 includes a device as described in any one of Examples 1 to 9, including or excluding optional features. In this example, the model includes a time-delayed neural network.

[0075] Example 11 is a method for speech recognition. The method includes: receiving an audio stream via a processor. The method further includes: detecting key phrases in the audio stream via the processor. The method further includes: dynamically adapting a model based on the detected key phrases via the processor. The method further includes: detecting sound queries following the key phrases in the audio stream via the adapted model via the processor.

[0076] Example 12 includes the method described in Example 11, including or excluding optional features. In this example, the adaptation model includes backpropagating the error at each time step back to the initial state.

[0077] Example 13 includes a method as described in any one of Examples 11 to 12, including or excluding optional features. In this example, adapting the model includes adjusting multiple weights of the model during backpropagation.

[0078] Example 14 includes a method as described in any one of Examples 11 to 13, including or excluding optional features. In this example, the adaptation model includes calculating the error by comparing the probability distribution of identified multi-phonemes of key phrases spoken by a particular speaker with an optimal distribution.

[0079] Example 15 includes a method as described in any one of Examples 11 to 14, including or excluding optional features. In this example, adapting the model includes adjusting the initial state of the model during backpropagation.

[0080] Example 16 includes a method as described in any one of Examples 11 to 15, including or excluding optional features. In this example, detecting key statements includes performing a forward pass on the model.

[0081] Example 17 includes a method as described in any one of Examples 11 to 16, including or excluding optional features. In this example, detecting key statements includes processing the audio stream via an ultra-low power wake-up statement detector.

[0082] Example 18 includes a method as described in any one of Examples 11 to 17, including or excluding optional features. In this example, the method includes generating a speech feature stream based on an audio stream. Key phrases are then detected based on the feature stream.

[0083] Example 19 includes a method as described in any one of Examples 11 to 18, including or excluding optional features. In this example, detecting a sound query includes generating a probability distribution over multiple phonemes.

[0084] Example 20 includes a method as described in any one of Examples 11 to 19, including or excluding optional features. In this example, detecting a sound query includes calculating a final letter sequence.

[0085] Example 21 is at least one computer-readable medium for speech recognition, wherein instructions are stored to guide a processor to receive an audio stream. The computer-readable medium also includes instructions for the guide processor to detect key phrases in the audio stream. The computer-readable medium further includes instructions for the guide processor to dynamically adapt a model based on the detected key phrases. The computer-readable medium also includes instructions for the guide processor to detect sound queries in the audio stream following the key phrases using the adapted model.

[0086] Example 22 includes a computer-readable medium as described in Example 21, including or excluding optional features. In this example, the computer-readable medium includes instructions for propagating errors at various time steps back to the initial state.

[0087] Example 23 includes a computer-readable medium as described in any one of Examples 21 to 22, including or excluding optional features. In this example, the computer-readable medium includes instructions for adjusting multiple weights of the model in a backward pass.

[0088] Example 24 includes a computer-readable medium as described in any one of Examples 21 to 23, including or excluding optional features. In this example, the computer-readable medium includes instructions for calculating an error by comparing the identified multi-phoneme probability distributions of key phrases spoken by a particular speaker with an optimal distribution.

[0089] Example 25 includes a computer-readable medium as described in any one of Examples 21 to 24, including or excluding optional features. In this example, the computer-readable medium includes instructions for adjusting the initial state of the model during backward pass.

[0090] Example 26 includes a computer-readable medium as described in any one of Examples 21 to 25, including or excluding optional features. In this example, the computer-readable medium includes instructions for performing a forward pass on the model to detect key statements.

[0091] Example 27 includes a computer-readable medium as described in any one of Examples 21 to 26, including or excluding optional features. In this example, the computer-readable medium includes instructions for processing an audio stream via an ultra-low power wake-up statement detector.

[0092] Example 28 includes a computer-readable medium as described in any one of Examples 21 to 27, including or excluding optional features. In this example, the computer-readable medium includes instructions for generating a speech feature stream based on an audio stream. Key phrases are detected based on the feature stream.

[0093] Example 29 includes a computer-readable medium as described in any one of Examples 21 to 28, including or excluding optional features. In this example, the computer-readable medium includes instructions for generating probability distributions over multiple phonemes.

[0094] Example 30 includes a computer-readable medium as described in any one of Examples 21 to 29, including or excluding optional features. In this example, the computer-readable medium includes instructions for calculating a final letter sequence to detect a sound query.

[0095] Example 31 is a system for speech recognition. The system includes an audio receiver for receiving an audio stream. The system also includes a key phrase detector for detecting key phrases in the audio stream. The system further includes a model adapter for dynamically adapting a model based on the detected key phrases. The system also includes a query recognizer for detecting sound queries in the audio stream that follow the key phrases using the adapted model.

[0096] Example 32 includes a system as described in Example 31, with or without optional features. In this example, the system includes a digital signal processor for detecting key statements.

[0097] Example 33 includes a system as described in any one of Examples 31 to 32, with or without optional features. In this example, the system includes a neural network for detecting key statements and adapting the model.

[0098] Example 34 includes a system as described in any one of Examples 31 to 33, with or without optional features. In this example, the system includes a dedicated ultra-low power voice wake-up unit for detecting key statements based on a speech feature stream.

[0099] Example 35 includes a system as described in any one of Examples 31 to 34, including or excluding optional features. In this example, the key statement includes a wake-up statement.

[0100] Example 36 includes a system as described in any one of Examples 31 to 35, with or without optional features. In this example, the system includes a feature front-end for calculating a speech feature stream based on a received audio stream.

[0101] Example 37 includes a system as described in any one of Examples 31 to 36, with or without optional features. In this example, the model includes an acoustic model for generating probability distributions over multiple phonemes.

[0102] Example 38 includes a system as described in any one of Examples 31 to 37, with or without optional features. In this example, the model includes a language model for computing the final letter sequence.

[0103] Example 39 includes a system as described in any one of Examples 31 to 38, with or without optional features. In this example, the model includes a recurrent neural network.

[0104] Example 40 includes a system as described in any one of Examples 31 to 39, with or without optional features. In this example, the model includes a time-delayed neural network.

[0105] Example 41 is a system for speech recognition. The system includes means for receiving an audio stream. The system also includes means for detecting key phrases in the audio stream. The system further includes means for dynamically adapting a model based on the detected key phrases. The system also includes means for detecting sound queries following the key phrases in the audio stream using the adapted model.

[0106] Example 42 includes a system like that of Example 41, with or without optional features. In this example, the system includes means for detecting key statements.

[0107] Example 43 includes a system as described in any one of Examples 41 to 42, with or without optional features. In this example, the system includes means for detecting key statements and adapting the model.

[0108] Example 44 includes a system as described in any one of Examples 41 to 43, with or without optional features. In this example, the system includes means for detecting key phrases based on a speech feature stream.

[0109] Example 45 includes a system as described in any one of Examples 41 to 44, with or without optional features. In this example, the key statement includes a wake-up statement.

[0110] Example 46 includes a system as described in any one of Examples 41 to 45, with or without optional features. In this example, the system includes means for calculating a speech feature stream based on a received audio stream.

[0111] Example 47 includes a system as described in any one of Examples 41 to 46, with or without optional features. In this example, the model includes an acoustic model for generating probability distributions over multiple phonemes.

[0112] Example 48 includes a system as described in any one of Examples 41 to 47, with or without optional features. In this example, the model includes a language model for computing the final letter sequence.

[0113] Example 49 includes a system as described in any one of Examples 41 to 48, with or without optional features. In this example, the model includes a recurrent neural network.

[0114] Example 50 includes a system as described in any one of Examples 41 to 49, with or without optional features. In this example, the model includes a time-delayed neural network.

[0115] Not all components, features, structures, characteristics, etc., described and illustrated herein need to be included in a particular aspect or one or more. For example, if the specification states that a component, feature, structure, or characteristic "may," "may," or "can" be included, then inclusion of that particular component, feature, structure, or characteristic is not required. If the specification or claims refer to an element "a" or "an," it does not mean that there is only one element. If the specification or claims refer to an "additional" element, it does not exclude the existence of more than one additional element.

[0116] It should be noted that although some aspects have been described with reference to specific implementations, other implementations are possible according to some aspects. Furthermore, the arrangement and / or order of circuit elements or other features shown in the figures and / or described herein need not be arranged in the specific manner shown and described. Many other arrangements are possible according to some aspects.

[0117] In each system shown in the figures, elements may, in some cases, have the same or different reference numerals to indicate that the elements represented may be different and / or similar. However, elements can be flexible enough to have different implementations and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which is referred to as the first element and which as the second element is arbitrary.

[0118] It should be understood that the details in the foregoing examples can be used anywhere in one or more aspects. For example, all the optional features of the computing device described above can also be implemented with respect to the methods described herein or any of the computer-readable media. Furthermore, although flowcharts and / or state diagrams may have been used to describe aspects herein, the technique is not limited to those diagrams or the corresponding descriptions herein. For example, the process does not need to move through each shown block or state or proceed in exactly the same order shown and described herein.

[0119] This technology is not limited to the specific details set forth herein. In fact, those skilled in the art who benefit from this disclosure will recognize that many other variations can be made to the foregoing description and figures within the scope of this technology. Therefore, the appended claims include any modifications to these claims that define the scope of this technology.

Claims

1. A non-transitory computer-readable medium storing instructions that, in response to being executed on a computing device, cause the computing device to perform an operation comprising the following steps: Generate speech features based on the received audio stream; Key sentence detection is performed on the audio stream based on the aforementioned speech features; The automatic speech recognition (ASR) model is adapted based on detected key phrases to generate an adapted ASR model; and Speech recognition is performed on the speech features using the adapted ASR model to detect sound queries in the audio stream.

2. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: The key statement detection is performed regardless of the speaker.

3. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: Identify the voice query in relation to the speaker.

4. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: Noise cancellation is applied to the audio stream.

5. The non-transitory computer-readable medium according to claim 4, wherein, The audio stream is captured by a microphone or microphone array.

6. The non-transitory computer-readable medium according to claim 1, wherein, The ASR model is a recurrent neural network.

7. The non-transitory computer-readable medium according to claim 1, wherein, The detected key statements are pre-selected statements.

8. A calculation method, comprising: Generate speech features based on the received audio stream; Key sentence detection is performed on the audio stream based on the aforementioned speech features; The automatic speech recognition (ASR) model is adapted based on detected key phrases to generate an adapted ASR model; and Speech recognition is performed on the speech features using the adapted ASR model to detect sound queries in the audio stream.

9. The calculation method according to claim 8 further includes performing the key statement detection independently of the speaker.

10. The calculation method of claim 8 further includes identifying the voice query in relation to the speaker.

11. The calculation method according to claim 8 further includes applying noise cancellation to the audio stream.

12. The calculation method according to claim 11, wherein, The audio stream is captured by a microphone or microphone array.

13. The calculation method according to claim 8, wherein, The ASR model is a recurrent neural network.

14. The calculation method according to claim 8, wherein, The detected key statements are pre-selected statements.

15. A computing system, comprising: A memory that stores instructions; and One or more processors, In response to being executed by the one or more processors, the instructions cause the one or more processors to perform the computation method of any one of claims 8-14.

16. A computing device comprising means for performing the computing method according to any one of claims 8-14.

17. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the computation method of any one of claims 8-14.

Images