Latency reduction for multi-level speech recognition

Abstract

Systems and techniques for processing audio samples are provided. For example, a process may include receiving one or more audio samples in a first audio frame, and determining a first keyword detection score for the first audio frame using a first keyword detection model. One or more audio samples in an additional audio frame may be received. Based on the first keyword detection score exceeding a first threshold, the first keyword detection model may be used to determine a keyword detection score for each of the additional audio frames. The respective keyword detection score for each of the additional audio frames may be compared to a second threshold greater than the first threshold. Processing the first audio frame and the additional audio frame using a second keyword detection model may be skipped based on the respective keyword detection score exceeding the second threshold.

Description

Technical Field

[0001] This application relates to speech recognition. For example, the described systems and techniques are used to reduce the latency in performing multi-level speech recognition, which is based on skipping one or more speech recognition levels using detection score information. Background Art

[0002] Electronic devices such as smart phones, tablet computers, wearable electronic devices, smart TVs, etc. have become increasingly popular among consumers. These devices can provide voice and / or data communication functions via wireless or wired networks. In addition, such electronic devices may include other features that provide a variety of functions designed to enhance user convenience. Electronic devices may include speech recognition functions for receiving voice commands from users. When a voice command from a user is received and recognized, this function allows the electronic device to perform functions associated with the voice command (e.g., via keywords). For example, the electronic device can activate a voice assistant application, play an audio file, or take a picture in response to a voice command from a user.

[0003] Speech recognition can be implemented as an "always-on" function in an electronic device to maximize its utility. These always-on functions require continuously running software and / or hardware resources, which in turn results in persistent power consumption. Mobile electronic devices, Internet of Things (IoT) devices, etc. are particularly sensitive to this always-on power requirement because it can shorten battery life and consume other limited resources of the system, such as processing power. Summary of the Invention

[0004] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Accordingly, the following summary should not be considered an exhaustive overview of all contemplated aspects, nor should it be considered to identify key or critical elements of all contemplated aspects or to delineate the scope associated with any particular aspect. Thus, the sole purpose of the following summary is to present some concepts related to one or more aspects of the mechanisms described herein in a concise form before the detailed description presented below.

[0005] Systems, methods, apparatuses, and computer-readable media for processing one or more audio samples are disclosed. According to at least one illustrative example, a method for processing one or more audio samples is provided. The method may include: receiving one or more audio samples in a first audio frame; determining a first keyword detection score for the first audio frame using a first keyword detection model; receiving one or more audio samples in an additional audio frame; based on the first keyword detection score exceeding a first threshold, determining a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; comparing each corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, where the second threshold is greater than the first threshold; and based on each corresponding keyword detection score exceeding the second threshold, skipping processing of the one or more audio samples in the additional audio frame using a second keyword detection model.

[0006] In another example, an apparatus for processing one or more audio samples is provided. The apparatus includes at least one memory and at least one processor, the at least one processor being coupled to the at least one memory. The at least one processor is configured to and may: receive one or more audio samples in a first audio frame; determine a first keyword detection score for the first audio frame using a first keyword detection model; receive one or more audio samples in an additional audio frame; based on the first keyword detection score exceeding a first threshold, determine a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; compare each corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, where the second threshold is greater than the first threshold; and based on each corresponding keyword detection score exceeding the second threshold, skip processing of the one or more audio samples in the additional audio frame using a second keyword detection model.

[0007] In another example, a non-transitory computer-readable medium of an apparatus is provided. The non-transitory computer-readable medium stores instructions thereon that, when executed by one or more processors, cause the one or more processors to: receive one or more audio samples in a first audio frame; determine a first keyword detection score for the first audio frame using a first keyword detection model; receive one or more audio samples in an additional audio frame; based on the first keyword detection score exceeding a first threshold, determine a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; compare each corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, where the second threshold is greater than the first threshold; and based on each corresponding keyword detection score exceeding the second threshold, skip processing of the one or more audio samples in the additional audio frame using a second keyword detection model.

[0008] In another example, an apparatus for processing one or more audio samples is provided. The apparatus includes: components for receiving one or more audio samples in a first audio frame; components for determining a first keyword detection score of the first audio frame using a first keyword detection model; components for receiving one or more audio samples in an additional audio frame; components for determining a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model based on the first keyword detection score exceeding a first threshold; components for comparing each corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, where the second threshold is greater than the first threshold; and components for skipping processing one or more audio samples in the additional audio frame using a second keyword detection model based on each corresponding keyword detection score exceeding the second threshold.

[0009] In some aspects, one or more of the apparatuses described herein are and / or include the following and / or are part of the following: an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device, a wireless communication device, a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device (e.g., an edge - or cloud - based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a vehicle acting as a server device, a network router, or other devices acting as a server device), another device, or a combination thereof. In some aspects, the apparatus includes one camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above - mentioned apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopic testers, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] The above aspects related to any one of method, apparatus, and computer - readable medium can be used individually or in any suitable combination.

[0011] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood with reference to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0012] The foregoing and other features and examples will become more apparent after referring to the following specification, claims, and drawings. Description of the Drawings

[0013] Examples of various specific implementations are described in detail below with reference to the following drawings:

[0014] Figure 1 is a block diagram illustrating an example speech recognition system according to some examples;

[0015] Figure 2 is a block diagram illustrating an example keyword detection system according to some examples;

[0016] Figure 3 is a block diagram illustrating an example feature generator according to some examples;

[0017] Figures 4A to 4C is a diagram illustrating an example of a neural network according to some examples;

[0018] Figure 5 is a block diagram illustrating an example of a deep convolutional network (DCN) according to some examples;

[0019] Figure 6 is a block diagram illustrating an example keyword detection system that uses detection score information according to some examples;

[0020] Figure 7 is a diagram illustrating an example of keyword detection scores over time for an example audio sample according to some examples;

[0021] Figure 8 is a flowchart illustrating an example of a process for processing one or more audio samples according to some examples; and

[0022] Figure 9 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. Detailed Description

[0023] Certain aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, specific details are set forth for purposes of explanation to provide a thorough understanding of the aspects and examples of the present application. However, it will be apparent that the various aspects and examples can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0024] The following description provides only example aspects and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the example aspects will provide those skilled in the art with a description that can be used to implement the example aspects. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0025] Voice recognition generally refers to the discrimination of human voices by an electronic device in order to perform a certain function. One type of voice recognition is keyword detection (e.g., wake word detection). Keyword detection refers to the technology by which a device detects specific vocabulary and responds. For example, many consumer electronics can utilize keyword detection to identify specific keywords to perform certain actions, such as "waking up" the device, querying information, and / or enabling the device to perform various other functions. Voice recognition can also be used for more complex functions, such as far-field voice recognition (e.g., from a mobile device placed across a room), user identification verification (e.g., via voice signature), voice recognition during other audio output (e.g., detecting voice commands while music is being played back on the device or detecting interrupt commands while a smart assistant is speaking), and voice interaction in a complex noise environment (such as inside a moving vehicle). These are just a few examples, and there may be many other examples.

[0026] Like various other processing tasks on an electronic device, voice recognition requires power and dedicated hardware and / or software to operate. Additionally, voice recognition can be implemented as an "always-on" function (e.g., where audio is continuously monitored for keyword detection) to maximize its utility for users of electronic devices with voice recognition capabilities. For plugged-in devices, the power usage of the always-on voice recognition function mainly concerns efficiency optimization; but for power-sensitive devices equipped with this function (e.g., battery-powered devices, mobile electronic devices, IoT devices, etc.), its power usage requires more attention. For example, the power usage of the always-on function may limit the runtime of such devices and reduce the capacity for other system processing requirements.

[0027] Speech recognition may include voice activity detection. For example, voice activity detection may refer to a computing device detecting human speech in order to perform a certain function. For example, keyword detection (e.g., also referred to as keyword recognition and / or keyword spotting (KWS)) is the task of detecting one or more keywords in an audio signal (e.g., an audio signal including human speech or uttered words). For example, keyword detection can be used to distinguish an activation phrase or a specific command from other speech and noise in the audio signal. In some cases, a keyword detection system can be targeted at or utilized by edge devices such as mobile phones and smart speakers. The detected keywords can include single words, compound words, phrases including multiple words, etc. In some cases, keyword detection can be performed based on a pre-determined keyword set and / or a user-defined keyword set. In some cases, user-defined keywords can include one or more adaptations, adjustments, etc. determined based on specific characteristics of a given user's speech or utterance.

[0028] Keyword detection can be performed on one or more audio data inputs (e.g., also referred to herein as "audio data", "audio signal", and / or "audio samples"). For example, the audio samples provided to a keyword detection system can be a streaming audio signal. In some examples, keyword detection can be performed on the streaming audio signal in real time. The streaming audio signal can be recorded by or obtained from a microphone associated with a computing device. Keyword detection can be performed locally or remotely. For example, keyword detection can be performed locally using one or more processors of the same computing device that collects or obtains the streaming audio signal. In some examples, keyword detection can be performed remotely by sending the streaming audio signal (or its representation) from a local computing device to a remote computing device (e.g., the local computing device records the audio signal but offloads the keyword detection processing task to the remote computing device). Performing keywords locally can contribute to a reduction in total latency or computation time but result in a decrease in accuracy. Performing keywords remotely can result in an increase in latency but contribute to an increase in accuracy.

[0029] For example, a local computing device (e.g., a smart phone) typically has lower computing power than a remote computing device (e.g., a cloud computing system) and can thus generate keyword detection results with lower accuracy or overall performance, especially when subject to time constraints associated with providing keyword detection results in real time or near real time. For example, a local computing device may implement a keyword detection model with lower complexity than a keyword detection model implemented on a remote computing device in order to provide real-time keyword detection results. Lower accuracy keyword detection results can include false positives (e.g., identifying keywords that do not actually exist), false negatives (e.g., failing to identify keywords that are present), and classification errors (e.g., identifying a first keyword as some other keyword).

[0030] However, performing keyword detection remotely can introduce communication latency that can offset the accuracy benefits associated with remote keyword detection. For example, remote keyword detection can introduce latency along the communication path from the local computing device to the remote computing device (e.g., the time to send a streaming audio signal or its representation to the remote computing device) and along the return communication path from the remote computing device to the local computing device (e.g., the time to send the keyword detection result from the remote computing device back to the local computing device).

[0031] In some cases, multiple levels can be used to perform keyword detection. For example, multi-level keyword detection can be used to minimize the power consumption associated with performing keyword detection on a power-sensitive device (e.g., minimizing the power consumption associated with always-on keyword detection performed by a battery-powered device such as a smart phone or other mobile computing device). In multi-level keyword detection, one or more levels can implement a low-complexity and low-latency keyword detection model, and one or more subsequent levels can implement a higher-complexity keyword detection model. For example, multi-level keyword detection can be performed as two-level keyword detection. In such an example, the first-level keyword detection model can be a low-complexity and low-latency keyword detection model. Based on the first-level generated keyword detection output (e.g., a keyword detection output with a confidence level greater than or equal to a first threshold), the second-level keyword detection model can be activated and used to process the same audio sample (e.g., the same audio sample that triggered the detection output of the first level).

[0032] The second-level keyword detection model can be provided as a relatively high-complexity keyword detection model (e.g., the first-level keyword detection model is provided as a relatively low-complexity keyword detection model). The performance of the second-level keyword detection model can be higher than that of the first-level keyword detection model. The second-level keyword detection model can be used to provide double confirmation of the keyword detection (e.g., by verifying or confirming the keyword detection of the first level) or to rule out the first-level keyword detection as a false positive (e.g., invalidate the keyword detection of the first level). However, performing multi-level keyword detection and / or using multiple different keyword detection models can also be considered as increasing the end-to-end system latency of the keyword detection system.

[0033] As mentioned above, in some examples, keyword detection is typically performed in real time (or near real time) to allow a user to interact with one or more computing devices. The latency between the time a user utters a keyword (e.g., an activation phrase or a specific command) and the time the computing device provides a corresponding response or action can be an important factor in the user's willingness to utilize spoken commands (e.g., spoken keywords). In some cases, a latency of several seconds may frustrate the user or otherwise prevent them from using spoken keywords. Thus, there is a need for improved keyword detection performance in local and / or remote keyword detection implementations, as both local and remote keyword detection implementations are generally time - constrained processes.

[0034] This disclosure describes systems, apparatuses, methods (also referred to as processes), and computer - readable media (collectively referred to herein as "systems and techniques") for providing a keyword detection system that can dynamically skip one or more keyword detection levels based on detection score information determined for a first keyword detection level. In some cases, the detection score information can be detection score information. In some examples, the systems and techniques can skip some (or all) of the levels in a set of one or more downstream keyword detection levels after the first keyword detection level based on determining that the detection score for the first keyword detection level is greater than or equal to one or more thresholds. In some cases, skipping one or more keyword detection levels after the first keyword detection level can reduce the end - to - end latency associated with the keyword detection system. Skipping one or more keyword detection levels after the first keyword detection level can additionally reduce the power consumption associated with the keyword detection system (e.g., can reduce the power consumption associated with performing keyword detection on one or more input audio samples).

[0035] In some examples, the systems and techniques can dynamically skip one or more keyword detection levels based on analyzing the corresponding detection score information associated with the first keyword detection level and one or more additional keyword detection levels (e.g., downstream keyword detection levels after the first keyword detection level). In some aspects, one or more thresholds can be determined based on analyzing the corresponding detection score information associated with different keyword detection levels of a multi - level keyword detection system, as will be described in more depth below.

[0036] In some cases, the system and techniques can dynamically skip one or more keyword detection levels by performing keyword detection based on one or more trailing frames of an audio signal. For example, a trailing frame can refer to an audio frame that occurs after a keyword detection score has exceeded a predetermined threshold. For example, a first keyword detection level can be associated with a first detection threshold. When the detection score of the current audio frame (e.g., included in a plurality of audio frames of an audio sample) is greater than the first detection threshold, the first keyword detection level can generate a keyword detection output indicating the detected keyword. Audio frames that occur after or are otherwise obtained after the current audio frame has a detection score exceeding the first detection threshold can be referred to as "trailing" frames. In some aspects, the system and techniques can determine the detection score information for some (or all) of the trailing frames after the first detection threshold of the first keyword detection level has been exceeded. The detection score information determined for the trailing frames can be analyzed against an additional threshold to determine whether an additional keyword detection level (e.g., the second keyword detection level of a two-level keyword detection system) can be skipped. In an illustrative example, an additional threshold can be determined such that for the determined threshold, skipping the keyword detection level will result in the same keyword detection decision as the first keyword detection level.

[0037] In some aspects, the system and techniques can implement voice activity detection (VAD) and / or audio context detection (ACD) machine learning models to determine additional classification information for an input audio sample provided to a multi-level keyword detection system. For example, a VAD or ACD model can be used to classify an input audio sample (or its audio frames) as speech or noise (or other non-speech audio signals). The speech or noise classification information can be provided as an additional input to the multi-level keyword detection system and used to dynamically determine whether one or more additional keyword detection levels can be skipped. For example, the speech or noise classification information can be used in combination with the first-level detection score information (e.g., as described above) to jointly determine whether the second level of the multi-level keyword detection system can be dynamically skipped.

[0038] By skipping one or more keyword detection levels after the first keyword detection level, the end-to-end latency associated with the keyword detection system can be reduced. Skipping one or more keyword detection levels after the first keyword detection level can also reduce the power consumption associated with the keyword detection system (e.g., can reduce the power consumption associated with performing keyword detection on one or more input audio samples).

[0039] Aspects of the present disclosure will be described with reference to the figures.

[0040] Figure 1Illustrates an example embodiment of a system-on-chip (SoC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), latencies, frequency bin information, task information, and other information may be stored in a memory block associated with the neural processing unit (NPU) 108, stored in a memory block associated with the CPU 102, stored in a memory block associated with the graphics processing unit (GPU) 104, stored in a memory block associated with the digital signal processor (DSP) 106, stored in memory block 118, and / or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from memory block 118.

[0041] The SoC 100 may also include additional processing blocks customized for specific functions, such as the GPU 104, the DSP 106, the connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation long-term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and the multimedia processor 112 that may detect and recognize gestures, speech, and / or other interactive user actions or inputs, for example. In one embodiment, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a keyword detection system 120. In some examples, the sensor processor 114 may be associated with or connected to one or more sensors to provide sensor input to the sensor processor 114. For example, the one or more sensors and the sensor processor 114 may be provided in the same computing device, coupled to the same computing device, or otherwise associated with the same computing device.

[0042] In some examples, one or more sensors may include one or more microphones for receiving sound (e.g., audio input), the sound including sound or audio input that can be used to perform keyword spotting (KWS), which can be regarded as a specific type of keyword detection. In some cases, the sound or audio input received by one or more microphones (and / or other sensors) may be digitized into data packets for analysis and / or transmission. The audio input may include ambient sound near a computing device associated with the SoC 100 and / or may include speech from a user of the computing device associated with the SoC 100. In some cases, the computing device associated with the SoC 100 may additionally or alternatively be communicatively coupled to one or more peripheral devices (not shown) and / or be configured to communicate with one or more remote computing devices or external resources, e.g., using a wireless transceiver and a communication network such as a cellular communication network.

[0043] The SoC 100, DSP 106, NPU 108, and / or keyword detection system 120 may be configured to perform audio signal processing. For example, the keyword detection system 120 may be configured to perform the steps of KWS. As another example, one or more parts of the steps for voice KWS, such as feature generation, may be performed by the keyword detection system 120, while the DSP 106 / NPU 108 performs other steps, such as steps using one or more machine learning networks and / or machine learning techniques according to aspects of the present disclosure and as described herein.

[0044] Figure 2 An example keyword detection first stage 200 according to aspects of the present disclosure is illustrated. The keyword detection first stage 200 receives an audio signal (e.g., pulse code modulation (PCM) audio data from an analog microphone, pulse density modulation (PDM) high-definition audio from a digital microphone, etc.) from an audio source 202 in an electronic system. For example, the audio signal may be generated by one or more microphones of an electronic device, such as a mobile electronic device, a smart home device, an Internet of Things (IoT) device, or other edge processing device. In some cases, the audio signal may be received substantially in real time.

[0045] In some cases, certain devices (such as relatively low-power (e.g., battery-powered) devices) may include a two-stage speech recognition system, where a first keyword detection stage (e.g., the first keyword detection stage 200) generates a keyword detection output that can be used to activate a second keyword detection stage (e.g., the second keyword detection stage 214). In multi-stage keyword detection, one or more stages may implement a low-complexity and low-latency keyword detection model, and one or more subsequent stages may implement a higher-complexity keyword detection model.

[0046] For example, the model associated with the first keyword detection stage 200 can be a low-complexity and low-latency keyword detection model. Based on the first stage 200 generating a keyword detection output (e.g., a keyword detection output having a detection score greater than or equal to a first threshold), the model associated with the second keyword detection stage 214 can be activated and used to process the same audio sample (e.g., the same audio sample that triggered the detection output of the first stage 200). The relatively high-complexity and / or higher-performance second-stage keyword detection model can be used to provide a double confirmation of the keyword detection (e.g., by verifying or confirming the keyword detection of the first stage) or to reject the first-stage keyword detection as a false positive (e.g., invalidate the keyword detection of the first stage).

[0047] In some cases, the first stage 200 of keyword detection can be implemented using relatively low-power circuits such as DSP, codec circuits, etc. When a keyword is detected, the second stage 214 can be activated, and the second stage can process more complex tasks such as more free-form word recognition, detecting commands, performing tasks, etc. In some cases, the second stage can be executed on relatively high-power circuits such as processors, GPUs, ML / AI processors, etc.

[0048] As Figure 2 illustrated, the received audio sample (e.g., the audio sample from the audio source 202) is processed by the feature generator 204 of the first stage 200 of keyword detection. The feature generator 204 can be, for example, a hardware-implemented Fourier transform such as a fast Fourier transform (FFT) function or circuit. The Fourier transform is generally a function for deconstructing the time-domain representation of a signal (such as the received audio signal) into a frequency-domain representation. The frequency-domain representation can include voltages or powers present at different frequencies in the received audio signal. In some cases, the feature generator 204 can generate a set of features such as feature vectors based on these representations. The set of features can be output to the keyword detector 208. Examples of the feature generator 204 will be described in more detail with respect to Figure 3 It is noted that other or additional forms of feature generation can be used in other aspects and examples.

[0049] The keyword detector 208 can use the keyword detection model 212 to determine whether the received audio signal includes a portion of a keyword. In some cases, the keyword detector 208 can accept dozens to hundreds of audio frames per second as input, and the keyword detector 208 can attempt to detect portions of keywords in the audio signal. In some cases, the keyword detection model 212 of the keyword detector 208 can be part of a multi-stage speech recognition system.

[0050] After keyword detector 208 determines that a keyword has been detected in the received audio signal, keyword detector 208 generates a signal for second stage 214. For example, the detected keyword may cause an application to start, or wake up another part of the electronic device (e.g., the screen, other processors, or other sensors), run a query locally or at a remote data service, perform additional speech recognition processing, etc. In some aspects, second stage 214 may receive an indication that a keyword has been detected, while in other aspects and / or examples, second stage 214 may receive additional information, such as information specific to the detected keyword, such as one or more detected keywords in the voice activity. It should be noted that there may be additional functionality (not shown) between keyword detector 208 and second stage 214, such as additional stages of keyword activity detection or analysis.

[0051] Figure 3 An example of a feature generator 300 (such as Figure 2 feature generator 204) in accordance with aspects of the present disclosure is depicted. It should be understood that many techniques may be used to generate feature vectors of audio, and feature generator 300 is merely a single example of a technique that may be used to generate feature vectors.

[0052] Feature generator 300 receives an audio signal at signal preprocessor 302. As above, the audio signal may be from an audio source of an electronic device (such as a microphone), such as audio source 202.

[0053] Signal preprocessor 302 may perform various preprocessing steps on the received audio signal. For example, signal preprocessor 302 may divide the audio signal into parallel audio signals and delay one of the signals by a predetermined amount of time to prepare for inputting the audio signal into the FFT circuit.

[0054] As another example, signal preprocessor 302 may perform a windowing function, such as a Hamming, Hann, Blackman-Harris, Kaiser-Bessel window function, or other sine-based window function, which may improve the performance of further processing stages (such as signal domain transformer 304). Generally, the windowing (or window) function therein may be used to reduce the magnitude of the discontinuity at the boundaries of each finite sequence of the received audio signal data to improve further processing.

[0055] As another example, signal preprocessor 302 may convert the audio signal data from parallel to serial, or vice versa, for further processing. The preprocessed audio signal generated by signal preprocessor 302 may be provided to signal domain transformer 304, which may transform the preprocessed audio signal from a first domain to a second domain, such as from the time domain to the frequency domain.

[0056] In some aspects, the signal domain transformer 304 implements a Fourier transform, such as a Fast Fourier Transform (FFT). For example, in some cases, the Fast Fourier Transform can be a 16-band (or frequency bin, channel, or point) FFT that generates a compact feature set that can be efficiently processed by the model. In some cases, compared to conventional single-channel processing (such as conventional hardware SNR threshold detection), the Fourier transform provides fine spectral domain information about the incoming audio signal. The result of the signal domain transformer 304 is a set of audio features, such as a set of voltages, powers, or energies per band in the transformed data.

[0057] This set of audio features can then be provided to the signal feature filter 306, which can reduce the size of the feature set in the audio feature data or compress the feature set. In some aspects, the signal feature filter 306 can discard certain features from the audio feature set, such as symmetric or redundant features from multiple bands of a multi-band FFT. Discarding this data reduces the overall size of the data stream for further processing and can be referred to as compressing the data stream.

[0058] For example, in some cases, since the audio signal is real, a 16-band FFT can include 8 symmetric or redundant bands after the power is squared. Thus, the signal feature filter 306 can filter out redundant or symmetric band information and output the audio feature vector 308. In some cases, the output of the signal feature filter can be compressed or otherwise processed before being output as the audio feature vector 308.

[0059] The audio feature vector 308 can be provided to the keyword detector for processing by a keyword detection model (such as the keyword detector 208 and the keyword detection model 212 shown Figure 2 ).

[0060] In some cases, a voice detection model (such as the keyword detection model 212) can execute on the SoC 100 and / or its components (such as Figure 1 the DSP 106 and / or the NPU 108). In some cases, the voice detection model can be a machine learning model or system.

[0061] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without using explicit instructions. An example of an ML system is a neural network (also known as an artificial neural network), which can include a group of interconnected artificial neurons (e.g., neuron models). Neural networks can be used in various applications and / or devices, such as speech analysis, audio signal analysis, image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0062] Individual nodes in a neural network can simulate biological neurons by obtaining input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values are associated with each vector and node in the network, and these values limit the way the input data relates to the output data. For example, the input data for each node can be multiplied by the corresponding weight value, and the products can be summed. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the result to produce the output signal or "output activation" (sometimes called a feature map or activation map) of the node. The weight values can initially be determined by an iterative flow of training data through the network (e.g., the weight values are established during a training phase where the network learns how to identify a particular class based on the characteristics of its typical input data).

[0063] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multi-layer perceptron (MLP) neural networks, transformer neural networks, etc. For example, a convolutional neural network (CNN) is a feedforward artificial neural network. A convolutional neural network can include a collection of artificial neurons, each of which has a receptive field (e.g., a spatially local region of the input space) and together tile an input space. An RNN works on the principle of saving the output of a layer and feeding that output back to the input to help predict the result of that layer. A GAN is a generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that could plausibly have come from the original dataset. A GAN can include two neural networks that operate together, including a generative neural network that generates the synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data can be fed into an input layer, and one or more hidden layers provide an abstraction level for the data. Predictions can then be made on the output layer based on the abstracted data.

[0064] Deep learning (DL) is an example of machine learning techniques and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNNs or CNNs, and utilize multiple layers. Using multiple layers in a deep neural network allows for the gradual extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input to the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to the third layer of artificial neurons, and so on. The layers located between the input and output of the entire deep neural network are typically referred to as hidden layers. The hidden layers learn (e.g., are trained) to transform the intermediate input from the previous layer into a slightly more abstract and composite representation that can be provided to the subsequent layer until the final or desired representation is obtained as the final output of the deep neural network.

[0065] As mentioned above, a neural network is an example of a machine learning system and can include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processing is performed by the hidden nodes of one or more hidden layers, and an output is produced through the output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). The feature map can include filters, kernels, etc. The nodes can include one or more weights, which are used to indicate the importance of the nodes in one or more of the layers. In some cases, a deep learning network can have a series of many hidden layers, where the earlier layers are used to determine simple and low-level characteristics of the input, and the later layers build a hierarchy of more complex and abstract characteristics.

[0066] Deep learning architectures can learn a hierarchy of features. For example, if presented with visual data, the first layer can learn to identify relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to identify spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to identify combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to identify common visual objects or spoken phrases. When applied to problems with a natural hierarchy, deep learning architectures can perform particularly well. For example, the classification of motorized vehicles can benefit from first learning to identify wheels, windshields, and other features. These features can be combined in different ways in higher layers to identify cars, trucks, and airplanes.

[0067] Figures 4A to 4CIllustrates an example neural network that can be used for keyword detection in accordance with aspects of the present disclosure. The neural network can be designed to have multiple connectivity patterns. In a feedforward network, information passes from lower layers to higher layers, where each neuron in a given layer communicates with neurons in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. The neural network can also have recurrent or feedback (also referred to as top-down) connections. In a recurrent connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. Recurrent architectures can help identify patterns that span more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are referred to as feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input.

[0068] In some cases, the connections between layers of a neural network can be fully connected or locally connected. Figure 4A Illustrates an example of a fully connected neural network 402. In the fully connected neural network 402, neurons in the first layer can communicate their outputs to each neuron in the second layer such that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 4B Illustrates an example of a locally connected neural network 404. In the locally connected neural network 404, neurons in the first layer can be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 404 can be configured such that each neuron in the layer will have the same or a similar connectivity pattern, but the connection strengths can have different values (e.g., 410, 412, 414, and 416). The locally connected connectivity pattern can result in spatially distinct receptive fields in higher layers because neurons in a given region of a higher layer can receive inputs that are tuned, through training, to the attributes of a restricted portion of the network's total input.

[0069] An example of a locally connected neural network is a convolutional neural network. Figure 4C Illustrates an example of a convolutional neural network 406. The convolutional neural network 406 can be configured such that the connection strengths associated with the inputs to each neuron in the second layer are shared (e.g., 408). Convolutional neural networks can be well-suited for problems where the spatial location of the input is meaningful.

[0070] Figure 5 Is a block diagram illustrating an example of a deep convolutional network (DCN) 550 in accordance with aspects of the present disclosure. The DCN 550 can include multiple different types of layers based on connectivity and weight sharing. As Figure 5As shown, the DCN 550 includes convolutional blocks 554A and 554B. Each of the convolutional blocks 554A and 554B can be configured with a convolutional layer (CONV) 556, a normalization layer (LNorm) 558, and a max pooling layer (MAX POOL) 560.

[0071] The convolutional layer 556 can include one or more convolutional filters, which can be applied to the input data 552 to generate a feature map. Although only two convolutional blocks 554A and 554B are shown, the present disclosure is not limited thereto, but any number of convolutional blocks (e.g., blocks 554A and 554B) can be included in the DCN 550 according to design preferences. The normalization layer 558 can normalize the output of the convolutional filters. For example, the normalization layer 558 can provide whitening or lateral inhibition. The max pooling layer 560 can provide spatially downsampled aggregation to achieve local invariance and dimensionality reduction.

[0072] For example, the parallel filter bank of the deep convolutional network can be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In some examples, the parallel filter bank can be loaded onto the DSP 106 or ISP116 of the SOC 100. Additionally, the DCN 550 can access other processing blocks that may be present on the SOC 100, such as the sensor processor 114 and the keyword detection system 120 dedicated to sensors and navigation, respectively.

[0073] The deep convolutional network 550 can also include one or more fully connected layers, such as layer 562A (labeled "FC1") and layer 562B (labeled "FC2"). The DCN 550 can also include a logistic regression (LR) layer 564. Between each layer 556, 558, 560, 562A, 562B, 564 of the DCN 550 are weights (not shown) to be updated. The output of each of these layers (e.g., 556, 558, 560, 562A, 562B, 564) can serve as the input to the subsequent layer among these layers (e.g., 556, 558, 560, 562A, 562B, 564) in the deep convolutional network 550 to learn a hierarchical feature representation from the input data 552 (e.g., images, audio, video, sensor data, and / or other input data) provided at the initial convolutional block 554A.

[0074] To adjust the weights, the learning algorithm can compute the gradient vector of the weights. The gradient can indicate the amount by which the error will increase or decrease if the weights are adjusted. At the top layer, the gradient can directly correspond to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In the lower layers, the gradient can depend on the values of the weights and the error gradients computed in the higher layers. The weights can then be adjusted to reduce the error. This way of adjusting the weights can be referred to as "backpropagation" because it involves a "backward pass" through the neural network.

[0075] In practice, the error gradient of the weights can be computed over a small number of examples such that the computed gradient is close to the true error gradient. This approximation method can be referred to as stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new inputs can be presented to the DCN, and a forward pass through the network can produce an output 422 that can be considered an inference or prediction of the DCN.

[0076] The output of the DCN 550 is a classification score 566 of the input data 552. The classification score 566 can be a probability or a set of probabilities, where the probability is the probability that the input data includes features from the set of features that the DCN 550 is trained to detect.

[0077] In some cases, an ML system or model can be used to analyze each audio frame to determine whether a voice command is likely to be present. For keyword detection, the output of the ML network (such as a probability) can be referred to as a frame score. The frame score indicates the likelihood that the frame includes one or more parts of a voice command (such as a keyword). As an example, in the case of keyword detection in response to the keyword "hey device", the first audio frame can have an audio signal that includes a sound corresponding to "he". The ML network should output a higher frame score for the first audio frame compared to another audio frame that does not have an audio signal that includes a sound corresponding to a part of "hey device". Although discussed in the context of an ML system herein, in some cases, non-ML techniques can be used to analyze the audio frame to generate a frame score and determine whether a voice command is likely to be present. For example, Gaussian mixture models (GMMs), hidden Markov models (HMMs) (GMM-HMMs), dynamic time warping (DTW), and / or other processes using Gaussian acoustic models and / or N-gram language models, such as phoneme likelihood estimation, Viterbi decoding, etc., can be used. These non-ML techniques can also be skipped based on the techniques discussed herein.

[0078] Generally, an ML system or model can be relatively complex and power-intensive to execute, and reducing the use of the ML system or model to help reduce the power consumption of the keyword detection system can be beneficial.

[0079] As previously noted, systems and techniques for providing a keyword detection system are described herein that can dynamically skip one or more keyword detection levels based on detection score information determined for a first keyword detection level. For example, the systems and techniques can skip some (or all) of a set of one or more downstream keyword detection levels after the first keyword detection level based on determining that the detection score for the first keyword detection level is greater than or equal to one or more thresholds. In some cases, skipping one or more keyword detection levels after the first keyword detection level can reduce the end-to-end latency associated with the keyword detection system. Skipping one or more keyword detection levels after the first keyword detection level can also reduce the power consumption associated with the keyword detection system (e.g., can reduce the power consumption associated with performing keyword detection on one or more input audio samples).

[0080] Figure 6 is a block diagram illustrating an example keyword detection system 600 for using detection score information to process one or more audio samples according to some examples. The keyword detection system 600 can be a multi-level keyword detection system and can dynamically skip one or more keyword detection levels included in the multiple keyword detection levels using detection score information. In some aspects, the keyword detection system 600 can be a two-level keyword detection system and can include a first keyword detection level 610 and a second keyword detection level 620. In one illustrative example, the first keyword detection level 610 can be the same as or similar to Figure 2 the first keyword detection level 200 of Figure 2 and / or the second keyword detection level 620 can be the same as or similar to

[0081] the second keyword detection level 214 of Figure 2 For example, the first keyword detection level 610 can receive one or more audio samples from an audio source 602 (e.g., in a manner that is the same as or similar to the manner described previously above for

[0082] the first keyword detection level 200 and the audio source 202). For example, the first keyword detection level 610 can receive PCM data obtained from a microphone as input.

[0083] In some aspects, the keyword detection score information generated as output by the first keyword detection stage 610 may be a numerical value indicating the keyword detection score (e.g., the confidence level of detecting a specific keyword for the current audio frame). In an illustrative example, the first-stage keyword detection score determined by the first keyword detection stage 610 may be compared with a first threshold.

[0084] For example, the first-stage keyword detection score may be compared with a threshold T associated with the first keyword detection stage 610 s1 for comparison. The threshold T s1 may be a pre-determined threshold, such as a pre-determined detection score threshold of the keyword detection model implemented by the first stage 610.

[0085] When the detection score generated and output by the first stage 610 is greater than the threshold T s1 the multi-stage keyword detection system 600 may dynamically determine whether to further process the current audio frame using the second keyword detection stage 620 (e.g., the second keyword detection stage may implement a keyword detection model with a higher complexity and accuracy than the model of the first stage 610) or whether the second keyword detection stage 620 can be skipped. In some examples, one or more audio frames (and / or other suitable audio data) may be provided to a post-processing stage 630. For example, the second keyword detection stage 620 may be skipped, and audio data including at least the current audio frame may be provided to the post-processing stage 630. In some examples, the audio data may additionally or alternatively include one or more audio frames before the current audio frame and / or one or more audio frames after the current audio frame provided to the post-processing stage 630 based on skipping the second keyword detection stage 620. Providing the audio data including the one or more audio frames (e.g., as described above) to the post-processing stage 630 may correspond to a positive keyword detection determination, where the post-processing stage 630 performs one or more operations based on the presence of the detected keyword. For example, the post-processing stage 630 may include one or more command stream operations, one or more response operations, one or more audio context detection (ACD) operations, etc.

[0086] In some cases, the systems and techniques described herein may use context information to determine or obtain one or more (or all) of the detection score thresholds. For example, audio context information (e.g., including audio context and / or other information generated as output using ACD operations) may be used to obtain some (or all) of the detection score thresholds from a lookup table or other pre-determined information storage. In an illustrative example, the first threshold T corresponding to the audio context information may be obtained from a lookup table s1 and / or the threshold The detection score threshold. In some aspects, the audio context information may indicate the specific context or environment in which the one or more audio samples were obtained (e.g., and / or one or more audio frames including the one or more audio samples), or be associated with the specific context or environment. For example, the audio context information may indicate that the audio sample or audio frame was obtained from a phone in a pocket context, running context, driving context, noisy environment context (e.g., in a sports venue, in a vehicle, or other road noise, etc.). In some aspects, the detection score threshold associated with the corresponding audio context and / or audio context information may be pre-determined. For example, a lookup table of different detection score thresholds corresponding to different audio contexts may be pre-determined, or otherwise generated offline from the audio keyword detection described herein. In some cases, the system and technology may use the pre-determined detection score threshold based on the real-time detection or determination of the audio context information. For example, one or more ACD operations may determine the audio context information associated with one or more audio samples (e.g., audio frames) in real time, and use the determined audio context information to obtain the corresponding detection score threshold from a pre-determined lookup table.

[0087] In an illustrative example, when the detection score output by the first stage 610 is greater than the threshold T s1 the multi-level keyword detection system 600 may perform a detection confidence analysis at the detection confidence analysis stage 615 to determine whether the second keyword detection stage 620 can be skipped. For example, the detection score output of the first stage 610 (e.g., the detection score determined by the keyword detection model of the first stage 610 for the current audio frame) may be compared with the threshold . The threshold may be a pre-determined threshold, as will be described in more depth below.

[0088] In some aspects, based on the detection score output of the first stage 610 exceeding the threshold , the second keyword detection stage 620 may be skipped, and the current audio frame may be processed using the post-processing stage 630 (e.g., corresponding to the "yes" output between the detection confidence analysis stage 615 and the post-processing stage 630 as shown in Figure 6 ). Skipping the second keyword detection stage 620 may correspond to detecting the keyword of the current audio frame.

[0089] Based on the detection score output of the first stage 610 not exceeding the threshold , the second keyword detection stage 620 is not skipped, but is activated and used for further analysis of the audio sample (e.g., corresponding to Figure 6"No" output between the detection confidence analysis stage 615 and the keyword detection second stage 620 as shown. For example, an audio sample may include the current frame of audio data (e.g., the frame corresponding to the output of the first-stage keyword detection score that is finally compared with the threshold ), and may additionally include one or more previous frames of the audio data before the current frame. The audio sample may additionally include one or more frames of subsequent audio data after the current frame. When the keyword detection second stage 620 is activated, the second keyword detection model implemented by the second stage 620 may be used to analyze the audio sample. If the keyword detection output of the second stage 620 exceeds the detection score threshold associated with the second stage 620 (e.g., the second-stage detection score threshold T s2 ), then the audio sample may be provided to the post-processing stage 630. If the keyword detection output of the second stage 620 does not exceed the detection score threshold associated with the second stage 620 (e.g., does not exceed T s2 ), then no keyword is detected, and the post-processing stage 630 is not utilized and is not otherwise activated.

[0090] In one illustrative example, the threshold associated with the detection confidence analysis stage 615 may be different from the above-mentioned first-stage detection score threshold T s1 . For example, the threshold may be greater than the first-stage detection score threshold T s1 . The threshold may additionally be different from the second-stage detection score threshold T s2 .

[0091] In one illustrative example, the multi-stage keyword detection system 600 may process an audio sample including multiple audio frames. Each audio frame may be provided as input to the first stage 610, which generates a first-stage detection score as output. If the first-stage detection score determined for a given audio frame (e.g., an audio frame among the multiple audio frames included in the audio sample) is less than the first-stage detection score threshold T s1 , then the multi-stage keyword detection system 600 determines that no keyword is detected for the current audio frame and waits for the next audio frame.

[0092] When the first-stage detection score of the first stage 610 is greater than the first-stage detection score threshold T s1 , the multi-stage keyword detection system 600 may proceed to the detection confidence analysis stage 615. For example, if the first-stage detection score of audio frame t > T s1 , then the multi-stage keyword detection system 600 may proceed to the detection confidence analysis stage 615 and compare the first-stage detection score of audio frame t with the threshold .

[0093] In an illustrative example, if the first - stage detection score of audio frame t > T s1 , the multi - stage keyword detection system 600 may use the first stage 610 and the detection confidence analysis stage 615 to process one or more trailing audio frames (e.g., audio frames after audio frame t). For example, once it is determined that the first - stage detection score of a given audio frame t > T s1 , the first stage 610 may be used to process subsequent audio frames t + 1, t + 2, …… (e.g., trailing audio frames), and the detection confidence analysis stage 615 may be used to compare the corresponding first - stage detection scores of each trailing audio frame with a threshold . In some aspects, based on the first - stage detection score of the current audio frame > T s1 , the detection confidence analysis stage 615 is activated for subsequent audio frames.

[0094] For example, based on the possibility that the first - stage detection scores of subsequent (e.g., trailing) audio frames may increase, the detection confidence analysis stage 615 may be used for subsequent audio frames after the initial first - stage detection score > T s1 . Figure 7 FIG. 700 is a diagram illustrating an example of keyword detection scores over time for an example audio sample 702 according to some examples.

[0095] Illustrated are a first set of detection scores 720 and a second set of detection scores 730. The first set of detection scores 720 may correspond to an audio sample with a relatively low signal - to - noise ratio (SNR) (e.g., such as when the example audio sample 702 is obtained in a noisy background environment). The second set of detection scores 730 may correspond to an audio sample with a relatively high SNR (e.g., such as when the example audio sample 702 is obtained in a low - noise or quiet background environment).

[0096] The first set of detection scores 720 and the second set of detection scores 730 may be detection scores determined for the audio sample 702 using the first keyword detection stage of the multi - stage keyword detection system. For example, the first set of detection scores 720 and the second set of detection scores 730 (respectively) may be detection scores determined using the first keyword detection stage 610 of Figure 6 and / or the first keyword detection stage 200 of Figure 2 .

[0097] Figure 7 The horizontal axis of s1 may be a time axis. For example, before time t1, no keyword is detected in either of the two sets of detection scores 720, 730. As illustrated, the first - stage detection scores (e.g., y - axis values) before time t1 are less than the first - stage detection score threshold T (e.g., indicated as the horizontal line 740). The threshold Figure 6associated with the detection confidence analysis stage 615) is also at Figure 7 depicted as a horizontal line 775 in >T s1 .

[0098] At time t1, the high SNR detection score 730 becomes greater than the first-stage detection threshold T s1 , as indicated at the keyword detection point 734. The high SNR detection score 730 can first exceed the first-stage detection threshold T s1 , and then exceed the threshold . For example, the first-stage detection threshold T s1 can have a value of 40, and the threshold can have a higher value, such as 75. Noisy (e.g., relatively low SNR) audio inputs can be associated with detection score values below the high SNR detection score 730 and can proceed to the second-stage keyword detection process. Relatively high SNR audio inputs can be associated with detection score values higher than the high SNR detection score 730 and / or higher than the threshold , and if the relatively high SNR detection score value exceeds the threshold , then the second-stage keyword detection process can be skipped. In some aspects, relatively low SNR audio inputs can be associated with relatively low (or lower) detection score values, and relatively high SNR audio inputs can be associated with relatively high (or higher) detection score values.

[0099] In some cases, the first-stage detection threshold T s1 and the threshold can be implemented using various values, where the value of the threshold is greater than the corresponding value of the first-stage detection threshold T s1 . In the above example where the first-stage detection threshold T s1 has a value of 40 and the threshold has a value of 75, the second-stage keyword detection and the first-stage keyword detection can make the same keyword detection decision for audio samples with detection scores greater than 75.

[0100] In some examples, the detection threshold can be given as a percentage (e.g., a percentage within the range [0, 100]). In some cases, the detection threshold can be a confidence percentage associated with keyword detection. In some examples, the first-stage detection threshold T s1 value of 40 can be associated with a detection confidence or detection probability of 40%. Larger values associated with the detection threshold (e.g., T s1 , , etc.) can be associated with detecting keywords with greater (e.g., better) confidence.

[0101] In some examples, a threshold may be determined based on a first threshold T s1 (e.g., also referred to as a second - level detection threshold or a second threshold). For example, as previously noted, a second threshold value may be determined such that the second threshold is greater than the first threshold T . In some cases, the value of the second threshold may be determined based on the value of the first threshold. In an illustrative example, the value of the second threshold may be equal to the value of the first threshold plus at least 80% of the value of the first threshold (e.g., s1 ≥1.8T ). s1

[0102] In some aspects, the first threshold T s1 and / or the second threshold may be determined based on signal - to - noise ratio (SNR) information. For example, the first threshold and / or the second threshold may be dynamically determined according to the SNR of one or more audio frames and / or audio samples. Audio data associated with a relatively high SNR (e.g., audio frames, audio samples, etc.) may be used to perform keyword detection with a relatively large detection score or confidence. Audio data associated with a relatively low SNR (e.g., noisy) may be used to perform keyword detection with a relatively low detection score or confidence. In an illustrative example, one or more (or both) of the first threshold and the second threshold may be determined based on one or more SNR measurements associated with audio data (e.g., audio frames, audio samples) processed by the systems and techniques described herein. For example, based on the fact that keyword detection is associated with greater confidence for high - SNR conditions relative to low - SNR conditions, the value of the first and / or second threshold for high - SNR audio data may be less than the corresponding value of the first and / or second threshold for low - SNR audio data. In some cases, the SNR associated with one or more audio samples in additional audio frames may be greater than the SNR associated with one or more audio frames in the first audio frame. The SNR associated with an audio frame may be determined as the average of the respective SNR values associated with each audio sample in one or more audio samples in the audio frame. For example, the SNR associated with an audio frame may be the average SNR of the audio samples in the audio frame.

[0103] As Figure 7 depicted in the example of, the high - SNR detection score 730 continues to increase after time t1 and reaches a peak detection score 738 at a subsequent time t2. The low - SNR detection score 720 is associated with a keyword detection point 724 later than time t1 and is shown to increase to a peak detection score 728 at the subsequent time t2.

[0104] ​In an illustrative example, audio frames processed after first - level keyword detection (e.g., audio frames processed after keyword detection point 734 with a high SNR score of 730, audio frames processed after keyword detection point 724 with a low SNR score of 720, etc.) can be referred to as trailing audio frames, as described above. For example, the corresponding keyword detection score > T can be determined based on the first - level keyword detection 610 using Figure 6 to identify keyword detection points 734 and 724. Subsequent trailing audio frames can be processed by the first - level keyword detection 610 of s1 and the detection confidence analysis stage 615 of Figure 6 can be used to analyze the corresponding keyword detection score (e.g., the keyword detection score generated by the first - level keyword detection 610) against a threshold Figure 6 .

[0105] In some aspects, the detection confidence analysis stage 615 can compare the corresponding first - level keyword detection score of each trailing frame with a threshold . If the first - level keyword detection score does not exceed the threshold , the detection confidence analysis stage 615 can further determine whether the first - level keyword detection score of the current frame has increased relative to one or more previously processed trailing frames. In an illustrative example, the detection confidence analysis stage 615 and a threshold can be used to perform trailing audio frame processing until the trailing frame is identified as having a first - level keyword detection score > (e.g., in this case, the second - level keyword processing 620 can be skipped and audio processing can proceed to the post - processing stage 630), or until a peak detection score is reached (e.g., the detection score of the current frame starts to decrease relative to the detection scores of previous frames).

[0106] For example, the high SNR detection score 730 exceeds the threshold at a subsequent keyword detection point 736 that occurs before the peak detection score 738. In this example, the keyword detection for the high SNR audio sample associated with the score 730 can be confirmed at keyword detection point 736, and the second - level keyword processing can be skipped. For example, at the subsequent keyword detection point 736, the detection confidence analysis stage 615 can be exited, the second - level keyword detection 620 can be skipped, and the audio sample can proceed directly to the Figure 6 post - processing stage 630, which can be triggered for the high SNR audio sample associated with the detection score 730 (e.g., without waiting to reach the peak detection score at 738).

[0107] The low SNR detection score 720 initially exceeds the first - level detection threshold T at detection point 724 s1The trailing frames processed by the detection confidence analysis stage 615 have an increased keyword detection score that increases to a peak detection score 728 at time t2. However, the low SNR detection score 720 determined for the trailing frames never exceeds the threshold . Based on detecting the peak detection score 728 when the low SNR score 720 does not exceed the threshold , the detection confidence analysis stage 615 can be exited and processing can proceed to the second-stage keyword processing (e.g., Figure 6 the "No" option depicted between the detection confidence analysis stage 615 and the keyword detection stage 620 in

[0108] As previously mentioned, the threshold for skipping the second-stage keyword detection processing can be greater than the first-stage keyword detection threshold T s1 . The threshold can be determined such that the second-stage keyword detection processing always performs the same detections as the first-stage keyword detection processing for a given value according to the threshold . For example, the threshold can be determined as the first-stage detection score value at which the second-stage keyword detection processing has a 100% agreement with the first stage (e.g., compared to other various percentage agreements). In some examples, such as under low-noise or high SNR conditions associated with obtaining an input audio sample provided to the multi-stage keyword detection system disclosed herein, the first-stage keyword detection score can be high enough to exceed the threshold for skipping the second-stage keyword detection processing . In other examples, such as when the user's accent fits well with the keyword detection model used to implement the first stage, the first-stage keyword detection score can additionally be considered high enough to exceed the threshold for skipping the second-stage keyword detection processing .

[0109] In some aspects, the threshold for skipping the second-stage keyword detection processing can be determined based on offline simulation, on-target computation, and / or using one or more machine learning models , as will be described in turn below

[0110] For example, the threshold can be determined based on offline simulation using a large enough dataset that covers different noise and SNR conditions associated with detecting one or more specific keywords. In an illustrative example, for the first stage of keyword detection (e.g., Figure 6 the first stage 610 of Figure 6The second level (620) determines or otherwise monitors the detection scores. For example, first-level keyword detection scores and second-level keyword detection scores can be obtained for a set of N different keyword detection examples obtained under different noise and SNR conditions. Among the N different keyword detection examples, The first-level detection score can be determined to correspond to a predefined percentage of consistency at the second keyword detection level. For example, The first-level detection score value corresponding to 100% consistency at the second keyword detection level can be determined. In other examples, The first-level detection score values corresponding to 99% consistency at the second keyword detection level, 95% consistency at the second keyword detection level, etc. can be determined. In an illustrative example, It can be determined such that for x*N keyword detection examples in a given dataset, D s1 =D s2 . Here, D s1 represents the first-level keyword detection determination, D s2 represents the second-level keyword detection determination, and x represents the predefined percentage of consistency (e.g., for 100% consistency, x = 1; for 99% consistency, x = 0.99; etc.) between the first-level keyword detection determination and the second-level keyword detection determination.

[0111] In another illustrative example, the threshold for skipping the second-level keyword detection process can be determined based on calculations on the target . For example, the dataset of the above N different keyword detection examples can be obtained by recording the first-level keyword detection scores and second-level keyword detection scores of various keyword detections performed using a specific device (e.g., a specific device implementing the systems and techniques for multi-level keyword detection with dynamically skippable levels disclosed herein). In such an example, the multi-level keyword detection system can be initialized without the value of the threshold , and then the value of the threshold can be determined at a later time after the dataset of N different keyword detection examples has been accumulated or otherwise obtained.

[0112] In another illustrative example, one or more machine learning models can be used to determine the threshold for skipping the second-level keyword detection process . These machine learning models can be used to determine the threshold offline, can be used to determine the threshold on the target (e.g., on a device implementing multi-level keyword detection with dynamically skippable levels), and / or utilize a combination of both. For example, the machine learning model can be implemented on the target and can be run periodically to calculate the threshold for skipping the second-level keyword detection process value

[0113] In some aspects, the system and techniques can implement voice activity detection (VAD) and / or audio context detection (ACD) machine learning models to determine additional classification information for an input audio sample provided to a multi-level keyword detection system. For example, a VAD or ACD model can be used to classify an input audio sample (or its audio frames) as speech or noise (or other non-speech audio signals). The speech or noise classification information can be provided as an additional input to the multi-level keyword detection system and used to dynamically determine whether one or more additional keyword detection levels can be skipped. For example, the speech or noise classification information can be used in combination with first-level detection score information (e.g., as described above) to jointly determine whether the second level of the multi-level keyword detection system can be dynamically skipped.

[0114] Figure 8 is a block diagram illustrating a process 800 for processing one or more audio samples. Although the example process 800 depicts a particular order of operations, the order can be varied without departing from the scope of the present disclosure. For example, some of the depicted operations can be performed in parallel or in a different order that does not substantially affect the functionality of the process 800. In other examples, different components of an example device or system implementing the process 800 can perform functions substantially simultaneously or in a particular order.

[0115] At block 802, the process 800 includes receiving one or more audio samples in a first audio frame. For example, one or more of the sensors 114 of Figure 1 , the DSP 106 of Figure 1 , the connectivity 110 of Figure 1 and / or Figure 1 the multimedia processor 112 of Figure 2 can be used to receive one or more audio samples in a first audio frame. In some examples, one or more audio samples in a first audio frame can be received from the audio source 202 of Figure 2 . In some cases, one or more audio samples in a first audio frame can be received using the audio signal of Figure 3 . In some cases, one or more audio samples in a first audio frame can be received in the input data 552 of Figure 5 . In some examples, one or more audio samples in a first audio frame can be received using the audio source 602 of Figure 6 . In some examples, one or more audio samples in a first audio frame can be received using the audio data 702 of Figure 7 .

[0116] At block 804, process 800 includes determining a first keyword detection score for a first audio frame using a first keyword detection model. For example, the first keyword detection model can be the same as or similar to the keyword detection model 212 included in Figure 2 the first level 200 of keyword detection. In some examples, the first keyword detection model can be included in Figure 6 the first level 610 of keyword detection. In some cases, the first keyword detection model is included in the first keyword detection level of a multi-level keyword detection system (such as Figure 2 the multi-level keyword detection system and / or Figure 6 the multi-level keyword detection system 600). In some cases, the multi-level keyword detection system is a two-level keyword detection system.

[0117] At block 806, process 800 includes receiving one or more additional audio samples in additional audio frames. In some cases, the additional audio frames and the first audio frame can be included in audio data, where the additional audio frames are after the first audio frame. For example, the additional audio frames and the first audio frame can be included in Figure 7 the audio data 702, where the additional audio frames are after the first audio frame. In some cases, the additional audio frames include one or more trailing audio frames, each trailing audio frame being after the first audio frame and having a corresponding keyword detection score that exceeds a first threshold. For example, each trailing audio frame can be after the first audio frame and can have a corresponding keyword detection score that exceeds Figure 7 the first threshold T s1 740.

[0118] At block 808, process 800 includes: based on the first keyword detection score exceeding the first threshold, determining a corresponding keyword detection score for each audio frame in the additional audio frames using the first keyword detection model. For example, the first threshold can be the same as or similar to the first threshold T Figure 7 depicted in. In some cases, the first threshold is a keyword detection threshold associated with the first level of keyword detection, and where the first level of keyword detection implements the first keyword detection model. For example, the first threshold can be a keyword detection threshold associated with s1 the first level 200 of keyword detection and / or Figure 2 the first level 610 of keyword detection and / or Figure 6 the first level 610 of keyword detection.

[0119] At block 810, process 800 includes: comparing each corresponding keyword detection score for each audio frame in the additional audio frames with a second threshold, where the second threshold is greater than the first threshold. For example, the second threshold can be the threshold Figure 7 depicted in 775 are the same or similar. In some cases, the second threshold may be determined based on multiple keyword detection scores determined using the first keyword detection model and corresponding multiple keyword detection scores determined using the second keyword detection model. For example, the second threshold may be determined as the keyword detection score of the first keyword detection model associated with a predetermined percentage of consistency between the first keyword detection model and the second keyword detection model. In some cases, the second keyword detection model may be implemented by a keyword detection second stage that is the same or similar to the keyword detection second stage Figure 2 of the keyword detection second stage 214 and / or Figure 6 of the keyword detection second stage 620.

[0120] In some examples, the predetermined percentage of consistency is 100%, where each keyword detection score determined using the first keyword detection model and exceeding the second threshold indicates keyword detection consistency between the first keyword detection model and the second keyword detection model.

[0121] At block 812, process 800 includes: based on each respective keyword detection score exceeding the second threshold, skipping processing of one or more audio samples in an additional audio frame using the second keyword detection model. For example, as noted above, the second keyword detection model may be implemented by a keyword detection second stage that is the same or similar to the keyword detection second stage Figure 2 of the keyword detection second stage 214 and / or Figure 6 of the keyword detection second stage 620.

[0122] In some cases, process 800 includes: based on the respective keyword detection score exceeding the second threshold, determining that a keyword has been detected for one or more audio samples in the additional audio frame. In some cases, one or more audio samples in the additional audio frame may be provided to a post-processing stage of the keyword detection system based on the respective keyword detection score exceeding the second threshold. For example, one or more audio samples in the additional audio frame may be provided to post-processing 630 included in Figure 6 the keyword detection system 600.

[0123] In some cases, the first keyword detection model is included in a first keyword detection stage of a multi-stage keyword detection system, and the second keyword detection model is included in a second keyword detection stage of the multi-stage keyword detection system. For example, the first keyword detection model may be included in the first keyword detection stage 610 of the multi-stage keyword detection system 600, and the second keyword detection model may be included in the second keyword detection stage 620 of the multi-stage keyword detection system 600. In some cases, the multi-stage keyword detection system may be the same as Figure 6 the first keyword detection stage 610 of the multi-stage keyword detection system 600, and the second keyword detection model may be included in Figure 6 the second keyword detection stage 620 of the multi-stage keyword detection system 600. In some cases, the multi-stage keyword detection system may be the same asFigure 6 a two - stage keyword detection system 600 that is the same as or similar to the two - stage keyword detection system.

[0124] In some cases, skipping the use of the second keyword detection model to process one or more audio samples in the additional audio frames includes using a machine learning model to classify the additional audio frames as including speech audio signals or including non - speech audio signals. Based on classifying the additional audio frames as including speech audio signals, process 800 can skip using the second keyword detection model to process one or more audio samples in the additional audio frames. One or more audio samples in the additional audio frames can be provided to a post - processing stage of the keyword detection system based on the corresponding keyword detection scores exceeding a second threshold. For example, one or more audio samples in the additional audio frames can be provided to Figure 6 the post - processing stage 630 of the keyword detection system 600.

[0125] In some cases, process 800 can include: using a deep neural network voice activity detection (DNN - VAD) machine learning model to classify the additional audio frames as including speech audio signals or including non - speech audio signals. In some examples, an audio context detection (ACD) machine learning model can be used to classify the additional audio frames as including speech audio signals or including non - speech audio signals. In some cases, process 800 can include: based on classifying the additional audio frames as including non - speech audio signals, skipping the use of the second keyword detection model to process one or more audio samples in the additional audio frames, and skipping the use of the post - processing stage to process one or more audio samples in the additional audio frames.

[0126] In some aspects, the processes described herein (e.g., process 800 and / or any other process described herein) can be executed by a computing device or apparatus. In one example, process 800 and / or other techniques or processes described herein can be executed by Figure 6 a system. In another example, process 800 and / or other techniques or processes described herein can be executed by Figure 9 the computing system 900 shown in. For example, a computing device having the computing device architecture of the computing system 900 shown in Figure 9 can implement the operations of process 800, and / or can implement one or more of the components and / or operations described herein with respect to Figures 1 to 7 any of the figures.

[0127] A computing device can include any suitable device, such as a mobile device (e.g., a mobile phone), an extended reality (XR) device (e.g., a virtual reality (VR), augmented reality (AR), or mixed reality (MR) headset, AR or MR glasses, etc.), a wearable device (e.g., a network-connected watch or other wearable device), a vehicle (e.g., an autonomous or semi-autonomous vehicle) or a computing system or device of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, a laptop computer, a network-connected television, a camera, and / or any other computing device having the resource capabilities to perform the processes described herein (including process 800, process 800, and / or any other process described herein). In some cases, a computing device or apparatus can include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device can include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface can be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0128] The components of a computing device can be implemented in circuitry. For example, the components can include electronic circuits or other electronic hardware, and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or can include computer software, firmware, or any combination thereof for performing the various operations described herein and / or can be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0129] Process 800 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0130] Additionally, process 800 and / or any of the other processes described herein can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware, or combinations thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0131] Figure 9 An example of a computing system 900 that can implement the various techniques described herein is shown. Components of the computing system 900 communicate with each other using connection 905. Connection 905 can be a physical connection via a bus or a direct connection into the processor 910, such as in a chipset architecture. Connection 905 can also be a virtual connection, a networked connection, or a logical connection.

[0132] In some cases, computing system 900 is a distributed system, where the functions described in this disclosure can be distributed within one data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent many such components, each performing some or all of the functions of the described component. In some examples, the components can be physical or virtual devices.

[0133] Example system 900 includes at least one processing unit (CPU or processor) 910 and connection 905 that couples various system components including system memory 915 (such as read-only memory (ROM) 920 and random access memory (RAM) 925) to the processor 910. Computing system 900 can include a cache 912 of high-speed memory that is directly connected to, close to, or integrated as part of the processor 910. In some cases, computing system 900 can copy data from memory 915 and / or storage device 930 to cache 912 for quick access by processor 910. In this way, the cache can provide a performance enhancement that avoids delays while the processor 910 waits for data. These modules and other modules can control or be configured to control processor 910 to perform various actions. Other computing device memory 915 can also be used. Memory 915 can include a variety of different types of memory with different performance characteristics.

[0134] The processor 910 may include any general-purpose processor and hardware services or software services, such as Service 1 932, Service 2 934, and Service 3 936 stored in the storage device 930, which are configured to control the processor 910 and a dedicated processor in which software instructions are incorporated into the actual processor design. The processor 910 may be substantially a complete stand-alone computing system that includes multiple cores or processors, buses, memory controllers, caches, etc. A multi-core processor may be symmetric or asymmetric.

[0135] To enable user interaction, the computing system 900 may further include an input device 945, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The computing system 900 may further include an output device 935, which may be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 900. The computing system 900 may include a communication interface 940, which generally may govern and manage user input and system output. The communication interface may execute or facilitate receiving and / or transmitting wired or wireless communications via a wired and / or wireless transceiver, including using an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, Apple ® Lightning ® port / plug, an Ethernet port / plug, a fiber optic port / plug, a dedicated wired port / plug, Bluetooth ® wireless signaling, Bluetooth ® low energy (BLE) wireless signaling, iBeacon ®Communication via wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.9 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), worldwide interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 900 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no restrictions on operating on any particular hardware arrangement, and thus the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0136] The storage device 930 can be a non-volatile and / or non-transitory and / or computer-readable memory device and can be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cassette, flash memory card, solid state memory device, digital versatile disc, cassette tape, floppy disk, flexible disk, hard disk, magnetic tape, magnetic stripe / strip, any other magnetic storage medium, flash memory, memristor memory, any other solid state memory, compact disc read only memory (CD-ROM) disc, rewritable compact disc (CD) disc, digital video disc (DVD) disc, Blu-ray disc, holographic disc, another optical medium, secure digital (SD) card, micro secure digital (microSD) card, memory stick ®Cards, smart card chips, Europay, MasterCard and Visa (EMV) chips, subscriber identity module (SIM) cards, mini / micro / nano / pico SIM cards, other integrated circuit (IC) chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), other memory chips or cartridges, and / or combinations thereof.

[0137] The storage device 930 may include software services (e.g., Service 1 932, Service 2 934, and Service 3 936, and / or other services), servers, services, etc., which, when the code defining such software is executed by the processor 910, cause the system to perform functions. In some aspects, the hardware services that perform specific functions may include software components for performing the functions stored in a computer-readable medium connected to the necessary hardware components such as the processor 910, the connection 905, the output device 935, etc.

[0138] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. The computer-readable medium may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. The computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a process, a function, a subroutine, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. The information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0139] In some examples, a computer-readable storage device, medium, and memory can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0140] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, one of ordinary skill in the art will understand that these aspects and examples can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks containing devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the aspects and examples in unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the aspects and examples.

[0141] The various aspects and examples may be described above as a process or method, which is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. When the operations of a process are completed, the process is terminated, but the process may have additional steps not included in the figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0142] The processes and methods according to the above examples can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions can include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used can be accessed through a network. The computer-executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, the information used, and / or the information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0143] Devices implementing the processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and can take any form factor among a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) for performing the necessary tasks can be stored in a computer-readable or machine-readable medium. The processor can execute the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein can also be embodied in peripheral devices or plug-in cards. By additional example, such functionality can also be implemented on a circuit board among different chips or different processes executed on a single device.

[0144] Instructions, the medium for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0145] In the above description, aspects of the present application are described with reference to their specific aspects and examples, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the illustrative aspects and examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be construed to cover such variations, unless limited by the prior art. The various features and aspects of the above applications can be used separately or jointly. Additionally, without departing from the broader spirit and scope of this specification, the aspects and examples can be utilized in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For purposes of illustration, the methods are described in a particular order. It should be understood that in alternative aspects and examples, the methods can be performed in an order different from that described.

[0146] One of ordinary skill in the art should understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein can be replaced with less than or equal to (“ ”) and greater than or equal to (“ ”) symbols, respectively.

[0147] In cases where a component is described as “configured to” perform certain operations, such a configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform the operations, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0148] The phrase "coupled to" means that any component is physically connected to another component directly or indirectly, and / or any component communicates with another component directly or indirectly (e.g., connected to another component through a wired or wireless connection and / or other suitable communication interfaces).

[0149] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0150] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0151] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized, at least in part, by a computer-readable data storage medium comprising program code, the program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the techniques may be realized, at least in part, by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0152] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0153] Exemplary aspects of this disclosure include:

[0154] Aspect 1. An apparatus for processing one or more audio samples, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to: receive one or more audio samples in a first audio frame; determine a first keyword detection score for the first audio frame using a first keyword detection model; receive one or more audio samples in an additional audio frame; based on the first keyword detection score exceeding a first threshold, determine a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; compare the corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, wherein the second threshold is greater than the first threshold; and based on the corresponding keyword detection score exceeding the second threshold, skip processing the first audio frame and the additional audio frame using a second keyword detection model.

[0155] Aspect 2. The apparatus according to aspect 1, wherein the at least one processor is further configured to: based on the corresponding keyword detection score exceeding the second threshold, determine that a keyword has been detected for the one or more audio samples in the additional audio frame.

[0156] Aspect 3. The apparatus according to aspect 2, wherein the at least one processor is further configured to: based on the corresponding keyword detection score exceeding the second threshold, provide the one or more audio samples in the additional audio frame to a post-processing stage of a keyword detection system.

[0157] Aspect 4. The apparatus according to any one of aspects 1 to 3, wherein: the first keyword detection model is included in a first keyword detection stage of a multi-stage keyword detection system; and the second keyword detection model is included in a second keyword detection stage of the multi-stage keyword detection system.

[0158] Aspect 5. The apparatus according to aspect 4, wherein the multi-stage keyword detection system is a two-stage keyword detection system.

[0159] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the additional audio frame and the first audio frame are included in audio data, and wherein the additional audio frame is after the first audio frame.

[0160] Aspect 7. The apparatus according to any one of aspects 1 to 6, wherein the additional audio frame includes one or more trailing audio frames, each trailing audio frame being after the first audio frame and having a corresponding keyword detection score exceeding the first threshold.

[0161] Aspect 8. The apparatus according to any one of aspects 1 to 7, wherein the first threshold is a keyword detection threshold associated with a first level of keyword detection, and wherein the first level of keyword detection implements the first keyword detection model.

[0162] Aspect 9. The apparatus according to any one of aspects 1 to 8, wherein, in order to skip processing the one or more audio samples in the additional audio frames using the second keyword detection model, the at least one processor is configured to: classify the additional audio frames as including a speech audio signal or including a non-speech audio signal using a machine learning model; based on classifying the additional audio frames as including a speech audio signal, skip processing the one or more audio samples in the additional audio frames using the second keyword detection model; and based on the respective keyword detection scores exceeding the second threshold, provide the one or more audio samples in the additional audio frames to a post-processing stage of the keyword detection system.

[0163] Aspect 10. The apparatus according to aspect 9, wherein the at least one processor is configured to: classify the additional audio frames as including a speech audio signal or including a non-speech audio signal using a deep neural network voice activity detection (DNN-VAD) machine learning model.

[0164] Aspect 11. The apparatus according to any one of aspects 9 to 10, wherein the at least one processor is configured to: classify the additional audio frames as including a speech audio signal or including a non-speech audio signal using an audio context detection (ACD) machine learning model.

[0165] Aspect 12. The apparatus according to any one of aspects 9 to 11, wherein the at least one processor is configured to: based on classifying the additional audio frames as including a non-speech audio signal, skip processing the one or more audio samples in the additional audio frames using the second keyword detection model, and skip processing the one or more audio samples in the additional audio frames using the post-processing stage.

[0166] Aspect 13. The apparatus according to any one of aspects 1 to 12, wherein the at least one processor is further configured to: determine the second threshold based on a plurality of keyword detection scores determined using the first keyword detection model and corresponding plurality of keyword detection scores determined using the second keyword detection model.

[0167] Aspect 14. The apparatus according to aspect 13, wherein the at least one processor is further configured to: determine the second threshold as the keyword detection score of the first keyword detection model associated with a predetermined percentage of consistency between the first keyword detection model and the second keyword detection model.

[0168] Aspect 15. The apparatus according to aspect 14, wherein the predetermined percentage of consistency is 100%, and each keyword detection score determined using the first keyword detection model and exceeding the second threshold indicates the keyword detection consistency between the first keyword detection model and the second keyword detection model.

[0169] Aspect 16. A method for processing one or more audio samples, the method comprising: receiving one or more audio samples in a first audio frame; determining a first keyword detection score of the first audio frame using a first keyword detection model; receiving one or more audio samples in an additional audio frame; based on the first keyword detection score exceeding a first threshold, determining a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; comparing the corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, wherein the second threshold is greater than the first threshold; and based on the corresponding keyword detection score exceeding the second threshold, skipping processing of the first audio sample and the additional audio frame using a second keyword detection model.

[0170] Aspect 17. The method according to aspect 16, the method further comprising: based on the corresponding keyword detection score exceeding the second threshold, determining that a keyword is detected for the one or more audio samples in the additional audio frame.

[0171] Aspect 18. The method according to aspect 17, the method further comprising: based on the corresponding keyword detection score exceeding the second threshold, providing the one or more audio samples in the additional audio frame to a post-processing stage of a keyword detection system.

[0172] Aspect 19. The method according to any one of aspects 16 to 18, wherein: the first keyword detection model is included in a first keyword detection stage of a multi-stage keyword detection system; and the second keyword detection model is included in a second keyword detection stage of the multi-stage keyword detection system.

[0173] Aspect 20. The method according to aspect 19, wherein the multi-stage keyword detection system is a two-stage keyword detection system.

[0174] Aspect 21. The method according to any one of aspects 16 to 20, wherein the additional audio frame and the first audio frame are included in audio data, and wherein the additional audio frame is after the first audio frame.

[0175] Aspect 22. The method according to any one of aspects 16 to 21, wherein the additional audio frame includes one or more trailing audio frames, each trailing audio frame being after the first audio frame and having a corresponding keyword detection score exceeding the first threshold.

[0176] Aspect 23. The method according to any one of aspects 16 to 22, wherein the first threshold is a keyword detection threshold associated with a first level of keyword detection, and wherein the first level of keyword detection implements the first keyword detection model.

[0177] Aspect 24. The method according to any one of aspects 16 to 23, wherein skipping processing of the one or more audio samples in the additional audio frame using the second keyword detection model includes: classifying the additional audio frame as including a speech audio signal or including a non-speech audio signal using a machine learning model; based on classifying the additional audio frame as including a speech audio signal, skipping processing of the one or more audio samples in the additional audio frame using the second keyword detection model; and based on the corresponding keyword detection score exceeding the second threshold, providing the one or more audio samples in the additional audio frame to a post-processing stage of the keyword detection system.

[0178] Aspect 25. The method according to aspect 24, the method further comprising: classifying the additional audio frame as including a speech audio signal or including a non-speech audio signal using a deep neural network voice activity detection (DNN-VAD) machine learning model.

[0179] Aspect 26. The method according to any one of aspects 24 to 25, the method further comprising: classifying the additional audio frame as including a speech audio signal or including a non-speech audio signal using an audio context detection (ACD) machine learning model.

[0180] Aspect 27. The method according to any one of aspects 24 to 26, the method further comprising: based on classifying the additional audio frame as including a non-speech audio signal, skipping processing of the one or more audio samples in the additional audio frame using the second keyword detection model and skipping processing of the one or more audio samples in the additional audio frame using the post-processing stage.

[0181] Aspect 28. The method according to any one of aspects 16 to 27, the method further comprising: determining the second threshold based on a plurality of keyword detection scores determined using the first keyword detection model and corresponding plurality of keyword detection scores determined using the second keyword detection model.

[0182] Aspect 29. The method according to aspect 28, the method further comprising: determining the second threshold as the keyword detection score of the first keyword detection model associated with a predetermined percentage of consistency between the first keyword detection model and the second keyword detection model.

[0183] Aspect 30. The method according to aspect 29, wherein the predetermined percentage of consistency is 100%, and each keyword detection score determined using the first keyword detection model and exceeding the second threshold indicates keyword detection consistency between the first keyword detection model and the second keyword detection model.

[0184] Aspect 31. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 1 to 15.

[0185] Aspect 32. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 16 to 30.

[0186] Aspect 33. An apparatus, the apparatus including one or more components for performing the operations according to any one of aspects 1 to 15.

[0187] Aspect 34. An apparatus, the apparatus including one or more components for performing the operations according to any one of aspects 16 to 30.

[0188] Aspect 35. The apparatus according to any one of aspects 1 to 15, wherein the second threshold is equal to the first threshold plus at least 80% of the first threshold.

[0189] Aspect 36. The apparatus according to any one of aspects 1 to 15 or 36, wherein a first signal-to-noise ratio (SNR) associated with the one or more audio samples in the additional audio frame is greater than a second SNR associated with the one or more audio samples in the first audio frame.

[0190] Aspect 37. The apparatus according to aspect 36, wherein: the first SNR is an average of corresponding SNRs associated with each of the one or more audio samples in the additional audio frame; and the second SNR is an average of corresponding SNRs associated with each of the one or more audio samples in the first audio frame.

[0191] Aspect 38. The apparatus according to any one of aspects 1 to 15 or 35 to 37, wherein the at least one processor is configured to: determine a value of the second threshold based on a value of the first threshold and a signal-to-noise ratio (SNR) associated with the additional audio frame.

[0192] Aspect 39. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 35 to 38.

[0193] Aspect 40. An apparatus including one or more components for performing the operations according to any one of aspects 35 to 38.

[0194] Aspect 41. The method according to any one of aspects 16 to 30, wherein the second threshold is equal to the first threshold plus at least 80% of the first threshold.

[0195] Aspect 42. The method according to any one of aspects 16 to 30 or 41, wherein a first signal-to-noise ratio (SNR) associated with the one or more audio samples in the additional audio frame is greater than a second SNR associated with the one or more audio samples in the first audio frame.

[0196] Aspect 43. The method according to aspect 42, wherein: the first SNR is an average of corresponding SNRs associated with each of the one or more audio samples in the additional audio frame; and the second SNR is an average of corresponding SNRs associated with each of the one or more audio samples in the first audio frame.

[0197] Aspect 44. The method according to any one of aspects 16 to 30 or 41 to 43, wherein the at least one processor is configured to: determine a value of the second threshold based on a value of the first threshold and a signal-to-noise ratio (SNR) associated with the additional audio frame.

[0198] Aspect 45. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 41 to 44.

[0199] Aspect 46. An apparatus, the apparatus comprising one or more components for performing the operations according to any one of aspects 41 to 44.

Claims

1. An apparatus for processing one or more audio samples, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: receive one or more audio samples in a first audio frame; determine a first keyword detection score for the first audio frame using a first keyword detection model; receive one or more audio samples in additional audio frames; based on the first keyword detection score exceeding a first threshold, determine a respective keyword detection score for each of the additional audio frames using the first keyword detection model; compare the respective keyword detection score for each of the additional audio frames with a second threshold, wherein the second threshold is greater than the first threshold; and based on the respective keyword detection score exceeding the second threshold, skip processing the first audio frame and the additional audio frames using a second keyword detection model.

2. The apparatus according to claim 1, wherein the at least one processor is further configured to: based on the respective keyword detection score exceeding the second threshold, determine that a keyword has been detected for the one or more audio samples in the additional audio frames.

3. The apparatus according to claim 2, wherein the at least one processor is further configured to: based on the respective keyword detection score exceeding the second threshold, provide the one or more audio samples in the additional audio frames to a post - processing stage of a keyword detection system.

4. The apparatus according to claim 1, wherein: the first keyword detection model is included in a first keyword detection stage of a multi - stage keyword detection system; and the second keyword detection model is included in a second keyword detection stage of the multi - stage keyword detection system.

5. The apparatus according to claim 4, wherein the multi - stage keyword detection system is a two - stage keyword detection system.

6. The apparatus according to claim 1, wherein the additional audio frames and the first audio frame are included in audio data, and wherein the additional audio frames are after the first audio frame.

7. The apparatus according to claim 1, wherein the additional audio frames include one or more trailing audio frames, each trailing audio frame being after the first audio frame and having a respective keyword detection score exceeding the first threshold.

8. The apparatus according to claim 1, wherein the first threshold is a keyword detection threshold associated with a first stage of keyword detection, and wherein the first stage of keyword detection implements the first keyword detection model.

9. The device according to claim 1, wherein To skip processing the one or more audio samples in the additional audio frames using the second keyword detection model, the at least one processor is configured to: classify the additional audio frames as including a speech audio signal or including a non - speech audio signal using a machine - learning model; based on classifying the additional audio frames as including a speech audio signal, skip processing the one or more audio samples in the additional audio frames using the second keyword detection model; and Based on the corresponding keyword detection score exceeding the second threshold, provide the one or more audio samples in the additional audio frame to a post-processing stage of the keyword detection system.

10. The apparatus according to claim 9, wherein the at least one processor is configured to: classify the additional audio frame as including a speech audio signal or including a non-speech audio signal using a deep neural network voice activity detection (DNN-VAD) machine learning model.

11. The apparatus according to claim 9, wherein the at least one processor is configured to: classify the additional audio frame as including a speech audio signal or including a non-speech audio signal using an audio context detection (ACD) machine learning model.

12. The apparatus according to claim 9, wherein the at least one processor is configured to: Based on classifying the additional audio frame as including a non-speech audio signal, skip processing the one or more audio samples in the additional audio frame using the second keyword detection model, and skip processing the one or more audio samples in the additional audio frame using the post-processing stage.

13. The apparatus according to claim 1, wherein the at least one processor is further configured to: Determine the second threshold based on a plurality of keyword detection scores determined using the first keyword detection model and corresponding plurality of keyword detection scores determined using the second keyword detection model.

14. The apparatus according to claim 13, wherein the at least one processor is further configured to: Determine the second threshold as the keyword detection score of the first keyword detection model associated with a predetermined percentage of consistency between the first keyword detection model and the second keyword detection model.

15. The apparatus according to claim 14, wherein the predetermined percentage of consistency is 100%, and each keyword detection score determined using the first keyword detection model and exceeding the second threshold indicates keyword detection consistency between the first keyword detection model and the second keyword detection model.

16. The apparatus according to claim 1, wherein the second threshold is equal to the first threshold plus at least 80% of the first threshold.

17. The apparatus according to claim 1, wherein a first signal-to-noise ratio (SNR) associated with the one or more audio samples in the additional audio frame is greater than a second SNR associated with the one or more audio samples in the first audio frame.

18. The apparatus according to claim 17, wherein: The first SNR is an average of corresponding SNRs associated with each audio sample in the one or more audio samples in the additional audio frame; and The second SNR is an average of corresponding SNRs associated with each audio sample in the one or more audio samples in the first audio frame.

19. The apparatus according to claim 1, wherein the at least one processor is configured to: Determine the value of the second threshold based on the value of the first threshold and the signal-to-noise ratio (SNR) associated with the additional audio frame.

20. A method for processing one or more audio samples, the method comprising: Receiving one or more audio samples in a first audio frame; Determining a first keyword detection score for the first audio frame using a first keyword detection model; Receiving one or more audio samples in an additional audio frame; Based on the first keyword detection score exceeding a first threshold, determining a corresponding keyword detection score for each audio frame in the additional audio frame using the first keyword detection model; Comparing the corresponding keyword detection score for each audio frame in the additional audio frame with a second threshold, wherein the second threshold is greater than the first threshold; And Based on the corresponding keyword detection score exceeding the second threshold, skipping processing the first audio frame and the additional audio frame using a second keyword detection model.

21. The method according to claim 20, the method further comprising: Based on the corresponding keyword detection score exceeding the second threshold, determining that a keyword is detected for the one or more audio samples in the additional audio frame.

22. The method according to claim 21, the method further comprising: Based on the corresponding keyword detection score exceeding the second threshold, providing the one or more audio samples in the additional audio frame to a post-processing stage of a keyword detection system.

23. The method according to claim 20, wherein: The first keyword detection model is included in a first keyword detection stage of a multi-stage keyword detection system; and The second keyword detection model is included in a second keyword detection stage of the multi-stage keyword detection system.

24. The method according to claim 20, wherein the additional audio frame and the first audio frame are included in audio data, and wherein the additional audio frame is after the first audio frame.

25. The method according to claim 20, wherein the additional audio frame includes one or more trailing audio frames, each trailing audio frame being after the first audio frame and having a corresponding keyword detection score exceeding the first threshold.

26. The method according to claim 20, wherein skipping processing the one or more audio samples in the additional audio frame using the second keyword detection model includes: Classifying the additional audio frame as including a speech audio signal or including a non-speech audio signal using a machine learning model; Based on classifying the additional audio frame as including a speech audio signal, skipping processing the one or more audio samples in the additional audio frame using the second keyword detection model; and Based on the corresponding keyword detection score exceeding the second threshold, providing the one or more audio samples in the additional audio frame to a post-processing stage of a keyword detection system.

27. The method according to claim 26, the method further comprising: Classify the additional audio frames as including speech audio signals or as including non-speech audio signals using a deep neural network voice activity detection (DNN-VAD) machine learning model or an audio context detection (ACD) machine learning model.

28. The method according to claim 26, the method further comprising: Based on classifying the additional audio frames as including non-speech audio signals, skip processing the one or more audio samples in the additional audio frames using the second keyword detection model and skip processing the one or more audio samples in the additional audio frames using the post-processing stage.

29. The method according to claim 20, the method further comprising: Determine the second threshold based on a plurality of keyword detection scores determined using the first keyword detection model and corresponding plurality of keyword detection scores determined using the second keyword detection model.

30. The method according to claim 29, the method further comprising: Determine the second threshold as the keyword detection score of the first keyword detection model associated with a predetermined percentage of consistency between the first keyword detection model and the second keyword detection model.

Images