Speaker-adaptive speech end detection for conversational AI applications

Abstract

In various instances, an end of speech (EOS) of an audio signal is determined based at least in part on a speech rate of a speaker, and for a segment of the audio signal, an EOS is indicated based at least in part on an EOS threshold determined based at least in part on the speech rate of the speaker.

Description

Technical Field

[0001] This application claims priority to U.S. Patent Application No. 17 / 162,907, filed on January 29, 2021, entitled "SPEAKER ADAPTIVE END OF SPEECH DETECTION FOR CONVERSATIONAL AI APPLICATIONS", the entire content of which is hereby incorporated by reference in its entirety and for all purposes.

Background Art

[0002] Audio data is captured for a variety of different purposes, such as providing voice control or speech-based interaction with various devices. Unfortunately, detecting the appropriate elements of speech within the captured audio can prove challenging, especially for devices with limited resources. Specifically, errors in detecting speech start and end by the user can lead to errors, inaccuracies, and delays in processing the audio data captured for a variety of different purposes. Previous solutions risk poor results and high latency in speech end detection.

Summary of the Invention

Means for Solving the Problems

[0003] Various techniques are described while referring to the drawings.

Brief Description of the Drawings

[0004] [Figure 1] A diagram showing a system for analyzing audio data according to at least one embodiment. [Figure 2] A diagram showing an audio processing pipeline according to at least one embodiment. [Figure 3A] A diagram showing an approach to speech end detection according to at least one embodiment. [Figure 3B] This figure shows an approach to speech end detection, based on at least one embodiment. [Figure 4] This figure shows a process for analyzing speech within an audio signal, according to at least one embodiment. [Figure 5] This figure shows a process for determining speaker-adaptive speech termination and / or speech start thresholds, according to at least one embodiment. [Figure 6] This figure shows a process for flagging speech end based at least partially on a speaker-adaptive speech end threshold, according to at least one embodiment. [Figure 7] This figure shows an example of a parallel processing unit ("PPU") according to one embodiment. [Figure 8] This figure shows an example of a general processing cluster ("GPC") based on one implementation. [Figure 9] This figure shows an example of a memory partition unit according to one embodiment. [Figure 10] This figure shows an example of a streaming multiprocessor in one embodiment. [Figure 11] This figure shows a computer system in which various examples can be implemented based on a single embodiment. [Figure 12] This figure shows a distributed system according to at least one embodiment. [Figure 13] This figure shows a cloud computing environment according to at least one embodiment. [Figure 14] This figure shows a set of functional abstraction layers provided by a cloud computing environment, according to at least one embodiment. [Modes for carrying out the invention]

[0005] Embodiments of this disclosure relate to speaker-adaptive speech end (EOS) detection. A system and method are disclosed for detecting speech start (SOS) and EOS based at least partially on the speech rate of a particular speaker.

[0006] In contrast to conventional systems such as those described above, dynamically adapting SOS and EOS detection based at least partially on acquired information about a particular speaker (e.g., speech rate) reduces latency and improves the accuracy of SOS and EOS detection. This disclosure describes an automated speech recognition system for a speaker-adaptive EOS detector that is at least partially based on connectionist temporal classification (CTC). In various embodiments, the EOS threshold indicates the minimum duration of detected silence in audio data for flagging as EOS. In at least one embodiment, an EOS window length can be specified, which defines a sliding window size for EOS detection, and an EOS threshold that can be used to flag a portion of a transcript or time interval as EOS when the proportion of blank symbols within the sliding window meets or exceeds the EOS threshold. For example, with a window size of 500ms and an EOS threshold set to 95%, the CTC output string for a particular window containing at least 95% blank space should be flagged as EOS by the EOS detector, as described in more detail below.

[0007] To determine the EOS threshold for a particular speaker, the intervals between consecutive words (e.g., inter-word times) are calculated, at least partially based on the output of the CTC, as described in more detail below. In one example, the EOS threshold is calculated based on a set of inter-word times, at least partially based on the string of characters output by the CTC. When performing automated speech recognition with speaker-adaptive EOS, a certain number of words spoken by a particular speaker may be captured before estimating the speech rate of that speaker. In one example, a set of 25 inter-word times is obtained before calculating the speech rate.

[0008] As will be explained in more detail below, speech rate can be the degree of speed at which words are produced, and can be defined in a great many ways, including the average of a set of interword times, the maximum of a set of interword times, the minimum of a set of interword times, the variance of interword times, other functions of a set of interword times, and / or combinations thereof. In various embodiments, once the speech rate is determined, the EOS threshold is fitted or otherwise modified for a particular speaker. Similarly, speech rate can be used to fit or otherwise modify the SOS threshold. In various embodiments, the EOS threshold and / or SOS threshold are modified periodically or aperiodicly, at least in part, based on the speech rate.

[0009] Once the EOS and SOS thresholds for a particular speaker are determined, in various embodiments, the EOS and SOS thresholds are used to determine the SOS and EOS of the audio data acquired as part of the automated speech recognition process. As described in more detail below, the input audio may be processed by various components (e.g., feature extractors and neuroacoustic models) before being input to the CTC. In various embodiments, the CTC provides a probability distribution of possible characters, or a sample contained within this audio frame or time step of the audio input. In one example, these characters may include any appropriate alphanumeric characters and one or more special characters, such as blanks, to potentially represent time steps or audio frames where no other characters are detected. In various embodiments, the most likely characters are selected from the set of possible characters, at least in part on the probability distribution indicated in the CTC, in order to determine a string of characters to be processed by the EOS detector.

[0010] In an embodiment, the EOS detector determines the proportion of blank characters in a string of characters for a specific window within a sliding window. In one example, for a set of time steps in a string of characters, the EOS detector determines whether the current state is a speech state (e.g., whether the current character indicates speech) or a non-speech state (e.g., whether the current state indicates blank and / or silence). The EOS detector may then determine whether the proportion of blank characters (e.g., non-speech state) satisfies an EOS threshold. For example, if the EOS threshold is met, EOS may be flagged, and the decoder generates a transcript associated with the audio input, as described in more detail below.

[0011] In at least one embodiment, a microphone 108 of an audio capture device 106 can capture an audio signal, which may include speech 104 uttered by at least one person 102. In various embodiments, the microphone 108 includes a device for converting sound waves into variations of electrical energy, which can then be amplified, transmitted, or recorded for use in various embodiments described in more detail below. In one example, the microphone 108 is included as part of a headset worn by a user 102, where the output of the microphone 108 is provided as an input to the audio capture device 106. In another example, the microphone 108 is part of an audio capture device 106 located away from the user 102 (for example, integrated with the audio capture device 106), where the audio capture device 106 may include or be part of a computing device, an Internet of Things (IoT) device, a smart speaker, a mobile phone, a computer, or other device capable of capturing or receiving audio signals or data. In various embodiments, the audio capture device 106 is integrated into another device, such as a navigation system, a vehicle, a security camera, or other system.

[0012] Furthermore, the audio capture device 106 may include network and / or processing functions. In one example, the audio capture device 106 includes a networking interface (e.g., a Wi-Fi radio) for transmitting data to the audio analyzer 112. In various embodiments, the audio analyzer 112 includes executable code or other instructions (e.g., source code stored in memory) that, when executed by the processing of the computing device, cause the computing device to perform various operations described in this disclosure, such as generating a transcript 116 based at least partially on the speech 104. For example, the audio capture device 106 includes memory and a processor used to execute the audio analyzer 112 or to perform other operations described in this disclosure. In yet another embodiment, the audio analyzer 112 is provided as a service by a computing resource service provider (e.g., a cloud computing service provider). In such embodiments, the audio analyzer 112 is executed by one or more servers (e.g., using virtual machines or container instances) in a data center to perform the operations described in this disclosure. In one example, an audio capture device 106 transmits one or more audio signals 110 over a network to a service endpoint associated with an audio analyzer 112.

[0013] In at least one embodiment, speech 104 uttered by user 102 may be speech that is transcribed or otherwise converted for any of several suitable purposes. For example, speech 104 is transcribed for use by a virtual assistant or other application. In another embodiment, speech 104 is parsed and converted into text commands that can be used to provide instructions or requests to at least one voice-controllable device 118. In at least one embodiment, the voice-controllable or voice-activated device 118 may include a device capable of performing physical actions, such as a vehicle, or a computing device containing software capable of executing instructions corresponding to a determined voice command or request.

[0014] In at least one embodiment, the audio capture device 106 can capture one or more audio signals 110 using a microphone 108, and the audio signals 110 can be communicated to an audio parser 112. As described above, the audio parser 112 can be on the same device as the audio capture device 106, or on a separate device such as a remote device available via a wireless connection or a remote server available via a network connection. Further, in various embodiments, the audio parser 112 includes a speech processing pipeline 114 that, as discussed in more detail below, uses the audio signal 110 as an input to analyze the audio signal 110 and generate one or more transcripts 116 or other text generated at least in part based on the audio signal. In various embodiments, the speech processing pipeline 114 includes executable code or other instructions (e.g., source code stored in memory) that, when executed by the processing of a computing device, cause the computing device to perform various operations described in this disclosure (e.g., using the audio signal 110 as an input to analyze the audio signal 110 and generate one or more transcripts 116). In at least one embodiment, the transcript 116 is provided to a voice controllable device 118, and the voice controllable device 118 can perform one or more actions based at least in part on the text contained in the transcript 116.

[0015] In various embodiments, the voice-controllable device 118 includes a command generator 120 that can parse the text contained in the transcript 116 and generate commands based at least in part on text executable by the controller 122 of the voice-controllable device 118 (or other application or component). In various embodiments, the command generator 120 and the controller 122 include executable code or other instructions (e.g., source code stored in memory) that, when executed by the processing of a computing device (e.g., the voice-controllable device 118), cause the computing device to perform various operations described herein, such as generating commands based at least in part on the transcript 116 and causing the computing device to execute the commands. For example, the commands are provided to coordinate the operation of a device, robot, or vehicle. In another example, the commands are provided to return information through a search engine, database query, or other operation of an application. In yet another embodiment, the transcript 116 is instead stored for subsequent use or analysis (e.g., the transcript 116 is stored in a database). As described above, the voice-controllable device 118 may be integrated within the same device as the audio analyzer 112, and may also include the audio capture device 106, or alternatively, it may be on a separate device capable of communicating with other components described herein by at least one wired or wireless connection. In one example, the voice-controllable device 118 is registered with a computing resource service provider (implemented as a service of the computing resource service provider) to receive a transcript 116 from the audio analyzer 112 over a network.

[0016] In at least one embodiment, the speech processing pipeline 114 includes various components (e.g., software and / or hardware components that execute source code), such as those described in more detail below in relation to Figure 2. In various embodiments, the speech processing pipeline 114 determines the speech rate of user 102 based at least in part on the audio signal 110. As described in more detail below, in embodiments, once a sufficient number of inter-word times have been determined, the speech processing pipeline 114 utilizes a speech end (EOS) threshold modified at least in part on user 102's rate speech. As described in more detail below, the speech processing pipeline 114 may use a Connectionist Temporal Classification (CTC) function or other algorithm to generate a string of characters representing the captured speech 104 in the audio signal 110. In embodiments, the speech processing pipeline 114 then determines a set of inter-word intervals (e.g., the time between spoken words in speech 104) based at least in part on blank characters included in the string of characters. The set of word intervals can then be used as input to a function for modifying and / or replacing the EOS threshold. As described in more detail below, the EOS threshold is used by the EOS detector to flag EOS signaling, which in the embodiment can generate a transcript 116 and / or provide it to a voice-controllable device 118.

[0017] Figure 2 shows an example where input audio 202, captured by a microphone or otherwise obtained, is provided as input to a speech processing pipeline 200 to generate a transcript 214. For example, the input audio 202 is captured by an audio capture device 106 as described above in connection with FIG. 1. In various embodiments, the speech processing pipeline 200 and its components (e.g., feature extractor 204, neural acoustic model, CTC, EOC detector 210, and decoder 212) include executable code or other instructions (e.g., source code stored in memory) that cause a computing device (e.g., an audio capture device 106 as described above in connection with FIG. 1) to perform various operations as will be described in more detail below when executed by the processing of the computing device. Further, as described above, the transcript may include text or other information suitable for use by a voice controllable device 118 as described in connection with FIG. 1.

[0018] The components shown in FIG. 2 may be implemented on a single computing device or multiple computing devices including a cloud computing device as described in this disclosure. For example, a local device (e.g., a cell phone, smart speaker, smart TV, or other device capable of capturing audio) captures the input audio 202 and provides the input audio 202 to a speech recognition pipeline 200 that is executed as a service of a computing resource service provider as will be described in detail below. Overall, the components shown in FIG. 2 may be executed by a computing device, a clouding computing device, or a combination thereof. Overall, the operations described herein can be distributed for implementation among a set of devices in different ways according to different embodiments.

[0019] In the embodiment, input audio 202 is provided to a feature extractor 204. The feature extractor 204 includes at least one neural network trained to accept the input audio 202 as input and output features representing the content of the input audio 202, such that in various embodiments the input audio 202 may correspond to one or more patterns recognized within one or more audio frames of the input audio 202. In one example, the feature extractor 204 includes a deep convolutional neural network (CNN) trained to extract features from the input audio 202. In at least one embodiment, the feature extractor 204 can perform tasks such as signal normalization, windowing, and audio feature spectrogram generation (such as a Mel spectrogram).

[0020] In at least one embodiment, a set of frequency-domain features relating to a given audio frame of input audio 202 (e.g., represented by a Mel spectrogram) can be transmitted to a neuroacoustic model 206 or other model useful for automatic speech recognition. For example, the neuroacoustic model 206 can generate a representation of the relationship between the received audio signal (e.g., input audio 2020) and the phonemes or other linguistic units that constitute the speech. Furthermore, in various embodiments, the neuroacoustic model 206 is trained, at least in part, by learning to produce a statistical representation of the sounds that constitute individual words.

[0021] In at least one embodiment, the output of the neuroacoustic model 206 is provided to the Connectionist Temporal Classification (CTC) function 208 to provide a probability distribution of possible characters, or a sample contained within a specific audio frame or time step of the input audio 202, taking these extracted audio features into consideration. In various embodiments, the characters output by the neuroacoustic model 206 include any suitable alphanumeric characters (e.g., A-Z) and potentially one or more special characters, such as blanks, to represent time steps or audio frames where no other characters are detected. In other words, blank characters represent time steps or audio frames where no speech was detected (e.g., pauses between words, where the speaker was silent). In at least one embodiment, the neuroacoustic model 206 can be any suitable acoustic model, such as the Jasper model from NVIDIA Corporation or other deep convolutional neural networks (CNNs). For example, the Jasper network can be used in an automated speech recognition (ASR) pipeline (e.g., speech processing pipeline 200) that includes a deep CNN consisting of a series of convolutional layers, as well as batch normalization, ReLU, and dropout layers.

[0022] To train tasks such as speech recognition, in at least one embodiment, a dataset of audio clips and corresponding transcripts is provided to the speech processing pipeline 200 as described in Figure 2. However, in some embodiments, the speech processing pipeline 200 is executed without alignment between the audio clips and transcripts (e.g., without training the data). In at least one embodiment, such a dataset may make it difficult to train a DNN-based speech recognition model due to this lack of aligned data, at least partially. However, in various embodiments, the Connectionist Temporal Classification (CTC) function 208 can be used to attempt to compensate for this lack of aligned data. For example, audio sequence X[x1,x2,...x T ] and transcript sequence Y[y1,y2,...y T Considering this, the CTC function 208 can be used with the output from the neuroacoustic model 206 to obtain an accurate mapping between this audio sequence and the transcript sequence. In at least one embodiment, the CTC function 208 is used with a given audio frame x i all y i A probability distribution for a value is provided. The probability distribution output by the CTC function 208 for each audio frame of the input audio 202 is provided as input to the decoder 212 for generating the transcript 214 in various embodiments. Furthermore, in at least one embodiment, this probability distribution for each audio frame is also provided as input to the speech end (EOS) detector 210, which can flag a particular time step or audio frame as EOS.

[0023] In at least one embodiment, the EOS detector 210 can be used to detect speech start (SOS) and EOS segments for the output of an automated speech recognition (ASR) model based on the output of the CTC function 208. In the embodiment, the CTC function 208 utilizes a defined number of characters (e.g., 29 characters), where these characters include the 26 English characters (and / or digits or other characters for other languages), as well as a set of other characters that indicate different grammatical and / or functional aspects, respectively, such as a space character for word separation, an apostrophe, and a blank character used to indicate that no other characters were detected for a given audio frame. As described above, the blank character can represent silence, and the probability of a blank character can be used for EOS detection. In at least one embodiment, the EOS detector 210 can be integrated with the CTC function 208 such that no additional components are required in the speech processing pipeline 200.

[0024] In various embodiments, the EOS detector 210 or other components of the speech processing pipeline 200 determine, or otherwise modify, an EOS threshold based on the determined speaker's speech rate. The speech rate may be determined at least in part based on the input audio 202 and / or several different audio signals provided to the speech processing pipeline 200. Furthermore, in embodiments, the speech processing pipeline 200 includes an initial EOS threshold. In one example, before it is possible to determine the speech rate for a particular speaker, the EOS threshold is set to 500 blank characters (obtained from the CTC function 208) (e.g., not limited to) a sliding window of size X (e.g., 500 milliseconds). In other examples, the EOS threshold is expressed as a percentage of blank characters (e.g., 90%) of a sliding window of size X.

[0025] In various embodiments, once a sufficient number of interword intervals have been calculated, the EOS threshold is modified, at least partially, based on the values ​​of these interword intervals. In one embodiment, the number of interword intervals is fixed (e.g., 25 interword intervals are required to calculate the EOS threshold). In yet another embodiment, the number of interword intervals required before modifying or generating a new EOS threshold is determined dynamically. For example, if the speech rate, as represented by a set of interword intervals, is consistent (e.g., the variance between interword intervals is small), the EOS threshold is modified before reaching the fixed value described above. Similarly, when the variance is large, additional interword intervals may be collected beyond the fixed value described above.

[0026] In various embodiments, once a sufficient set of interword intervals is obtained (for example, at least partially based on the output of the CTC function 208), the initial EOS threshold may be modified or a new EOS threshold may be generated. In one example, the EOS threshold is set to the maximum value of the set of interword intervals (e.g., the longest amount of time between spoken words). Furthermore, in various embodiments, the EOS threshold is determined according to the set of interword intervals. For example, the EOS threshold may be calculated at least partially based on the median, variance, mode, or other function of the values ​​in the set of interword intervals. In at least one embodiment, an algorithm using these values ​​may be given by: interword_times=update_interword_times(ctc_output_string,interword_times)SOS_threshold,EOS_threshold=calculate_dynamic_threshold(interword_times) detect_speech_segments(ctc_output_string){ segment={} startcompar0 for each time step in string{ If (state == speech) { Detecting EOS using EOS window length and EOS_threshold. If(EOS){ State=non-speech segments.add(start,current_timestep) } } else if(state=non-speech){ Detect SOS using SOS window length and SOS_threshold. If(SOS){ state=speech start=current_timematch } } } }

[0027] Figures 3A and 3B show a portion of the speech processing pipeline 300, in which various components of the EOS detector 306 process the output of the CTC function 304. In at least one embodiment, the EOS detector 306 includes a greedy decoder 308, as well as a component 310 for determining speech start (SOS) and / or speech end (EOS) (e.g., for flagging). As described above in relation to Figure 2, the acoustic model output 302 is provided as input to the CTC function 304, which in turn outputs a probability distribution for the possible characters for each time step. For example, the output of the CTC function 304 includes a probability distribution expressed as nx 1 xc, where n is the number of time steps and c is the number of entries in the vocabulary. In at least one embodiment, the probability distribution data generated by the CTC function 304 is provided as input to the greedy decoder 308. In the embodiment, the greedy decoder 308 selects the most likely character from a set of possible characters (e.g., characters indicated in the probability distribution data) at least in part on a probability distribution (e.g., 1 xc). In one embodiment, the Argmax decoder is applied at each time step such that the output of the greedy decoder 308 is a string of n characters, where the selected character has the highest probability as indicated in the probability distribution data. In at least one embodiment, the output 350 of the greedy decoder 308 (e.g., the Argmax function) is shown in Figure 3B, where this output includes a string of alphanumeric characters as well as blank symbols, which are underlined in Figures 3A and 3B.

[0028] In at least one embodiment, the string shown in Figures 3A and 3B (e.g., output 350) is time-aligned with the corresponding audio input, at least in part, based on the output of the CTC function 304. As shown in Figure 3A, in the embodiment, the string is provided as input to component 310 to determine EOS and / or SOS. For example, once EOS or SOS is determined for output 350, the EOS detector 306 flags a time step for the decoder of the speech recognition pipeline, indicating that a transcript can be generated, as described above. In at least one embodiment, component 310 can also calculate a speech start (SOS) value or time. For example, the output of the EOS detector 306 includes the start and end times of a given speech segment of the input audio signal. In various embodiments, the segment (e.g., EOS segment) includes various parts of the audio signal, including time steps, audio frames, sliding windows, or other logical or physical parts of the audio signal.

[0029] In at least one embodiment, EOS and / or OS detection is performed using a sliding window 352 for the output 350 of the greedy decoder 308. For example, the sliding window 352 includes several time steps X (e.g., 25), where each time step represents a time interval (e.g., 20 milliseconds). As described above, the determination of EOS can be performed at least in part on the proportion of blank symbols (shown as underlined in Figures 3A and 3B) contained in the sliding window 352 for any time step and / or range of time steps. In one example, each time step can be classified as speech or non-speech (e.g., as indicated in the algorithm described above in relation to Figure 2), where a blank symbol represents a non-speech period. Furthermore, in this example, a time step classified as speech may have another non-blank symbol (e.g., an alphanumeric character) assigned to indicate speech for that time step. In an embodiment, the CTC function 304 returns the most likely character for each time step.

[0030] As shown in Figure 3B, the word interval 354 in the embodiment includes several time steps or audio frames represented by blank symbols between alphanumeric characters. In at least one embodiment, separate window lengths and thresholds (e.g., EOS threshold and SOS threshold) can be used to provide finer control over segment detection, including both SOS and EOS determination. For example, various parameters such as the SOS window length can specify the sliding window size for speech start detection, and the SOS threshold value can be used to specify that SOS can be flagged if the percentage of blank symbols within this sliding window falls below the SOS threshold.

[0031] In another example, an EOS window length can be specified, which defines the sliding window size for EOS detection, and an EOS threshold that can be used to flag EOS when the percentage of blank symbols within this sliding window meets or exceeds this threshold. In various embodiments, the EOS window size is set to 500 milliseconds and the EOS threshold is set to 95%, based at least in part on a set of word intervals. Furthermore, a set of word intervals is used in embodiments to modify the SOS sliding window size, SOS threshold, EOS sliding window size, and EOS threshold. In this form, the EOS detector 306 may allow for stricter thresholds for the start and end of speech to ensure that speech is not missed, where the inclusion of additional non-speech is not important and may cause delays in generating a response to the user. In at least one embodiment, an algorithm using these values ​​can be given by: For each time step If (speech started) { EOS detection using EOS window length and threshold. else{ Detect SOS using SOS window length and threshold. } }

[0032] Furthermore, in various embodiments, the acoustic model can be trained to improve EOS detection by detecting noise as silence or non-speech. In one example, this may involve generating training data containing different types and levels of noise so that character probability detection is more accurate in the presence of different types of noise (e.g., road noise, background conversation, construction work, wind, or other forms of sound not directly generated by the act of a speaker speaking). In at least one embodiment, this may involve mixing different types and levels of noise with a clean audio signal to generate training data. In such embodiments, the CTC function 304 is used during training to align audio files with these transcripts. Furthermore, in various embodiments, the output of the CTC function 304 is decoded to generate a final transcript.

[0033] Furthermore, the speech recognition pipeline described herein can be robust to challenging acoustic conditions. For example, the speech recognition pipeline can handle both extremes, including low-gain and noisy environments. In various embodiments, the speech recognition pipeline utilizes a simple and computationally efficient algorithm with little additional cost for the EOS detector, and thus the EOS detector can utilize the output from the neuroacoustic model and CTC function 304 that may already be present within the speech recognition pipeline 300. As a result, the EOS threshold is recalculated periodically or aperiodically in various embodiments. In one example, a set of interword intervals is computed sequentially, and the EOS threshold is recalculated and / or updated every 25 interword intervals. In at least one embodiment, the EOS detector can utilize a neuroacoustic model trained according to a target environment, such as a house, car, office, or retail store.

[0034] Referring here to Figures 4, 5, and 6, each block of Methods 400, 500, and 600 described herein includes computing processes that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor that executes instructions stored in memory. Each of the Methods can also be embodied as computer-usable instructions stored on a computer storage medium. The Methods may be provided, to name a few, as a standalone application, a service (standalone or in combination with another hosted service), or a hosted service, or as a plug-in to another application or service. Furthermore, Method 400 is described as an example of the speech recognition pipeline in Figure 2. Nevertheless, these Methods may be performed by any one system or any combination of systems, including but not limited to those described herein, as additional or alternative. Furthermore, the operation of the Methods described in Figures 4, 5, and 6 may be omitted, performed in different orders, performed in parallel, or in a combination of sequential and parallel operations. Overall, the operations described below in relation to Figures 4, 5, and 6 can be performed by a single device or multiple devices communicating over a network or other communication channel.

[0035] Figure 4 shows a method 400 for determining the end of a speech segment in an audio signal according to various embodiments of the present disclosure. As described above, method 400 is carried out in embodiments by the audio capture device 106 described above in relation to Figure 1. In other embodiments, other devices (e.g., cloud computing resources, virtual machines, virtual processors, edge computing devices, etc.) perform all or some of the operations described below. For example, the audio capture device records an audio signal and transmits the audio signal or a portion thereof (e.g., a downsampled or otherwise processed version of the audio signal) to a cloud computing device, virtual machine or processor, or edge computing device, etc., in order to perform all or some of the operations described above.

[0036] Method 400 includes capturing an audio signal in block 402. In an embodiment, the audio signal is captured using a microphone (e.g., an audio capture device) to capture an audio signal representing human speech. For example, a smart speaker captures an audio signal in response to an invocation word and transmits the captured audio signal to a speech processing service implemented by a computing resource service provider. In block 404, a set of audio features is extracted from the audio signal. For example, the features include a frequency spectrum. In block 406, the features extracted from the audio signal are analyzed using a trained acoustic model and a Connectionist Temporal Classification (CTC) function to determine the character probability for individual frames or time steps of this audio signal. In block 408, the character probabilities output by the CTC function are analyzed using one or more speech measures to calculate the start and / or end times of one or more speech segments contained within this audio signal. In at least one embodiment, this can be carried out using a speech end (EOS) detector that uses a speech measure including the length of the sliding window and / or the percentage of blank characters predicted within the sliding window to determine the start and end of speech. Furthermore, in various embodiments, the window size and percentage thresholds can vary depending on the start and end time calculations. In block 410, a decoder can be used to convert the character probabilities between the start and end times into transcripts of the corresponding speech segments. In at least one embodiment, these transcripts can then be stored or provided to an intended recipient, such as a voice control device that is configured to function according to commands contained within a given speech segment.In the embodiment, blocks 404 to 410 are carried out by a speech processing service implemented by a computing resource service provider.

[0037] Figure 5 is a flowchart illustrating Method 500 for determining an End of Speech (EOS) threshold based at least partially on the speech rate of a particular speaker, according to several embodiments of the present disclosure. In some embodiments, Method 500 is implemented by an audio capture device, as described herein. In yet another embodiment, Method 500 is implemented as a component of a speech processing service implemented by a computing resource service provider. For example, a computing resource service provider's service uses Method 500 to determine an EOS threshold and provides and / or updates an EOS threshold for an audio capture device so that the audio capture device can improve its operation and EOS detection. Returning to Figure 5, Method 500 includes, in block 502, obtaining a string of characters. In various embodiments, a Connectionist Temporal Classification (CTC) function generates a probability distribution of characters as an output, based at least partially on the output of an acoustic model. For example, as described above, the CTC function takes into account the input features of the audio signal at each time step and outputs a probability distribution for the possible characters at each time step. Furthermore, as described above, components of the speech recognition pipeline (e.g., a greedy decoder) can use algorithms to generate character strings based at least partially on the probability distribution output by the CTC.

[0038] In block 504, a system implementing method 500 determines sets of intervals between words based at least in part on a string of characters. In various embodiments, the characters include a predetermined alphabet where an underline or other symbol represents a portion of an audio signal without speech. In other words, an audio signal at a given time step represented by blank symbols may contain sounds (e.g., background noise) but not speech sounds (e.g., spoken words). In embodiments, multiple consecutive blank symbols between words represented by a string of alphanumeric characters output by the CTC are counted as intervals between words. As described above, in embodiments, the number of intervals between words (e.g., 25) is collected before determining the EOS threshold. Still, in yet another embodiment, it is possible to determine the EOS threshold once the first interval between words has been determined. In such embodiments, the EOS threshold may be modified by subsequent intervals between words.

[0039] In block 506, the system implementing method 500 determines an EOS threshold based at least in part on a set of interword intervals. In various embodiments, the EOS threshold depends on a value in the set of interword intervals. In one example, the EOS threshold is set to the midpoint of the set of interword intervals. In yet another example, the EOS threshold is set to the maximum value in the set of interword intervals. Once the EOS threshold is determined, in various embodiments, the speech processing pipeline replaces the initial EOS with the EOS determined by the system implementing method 500. In various embodiments, method 500 is used to determine a speech start (SOS) threshold.

[0040] Figure 6 is a flowchart illustrating a method 600 for determining the end of speech (EOS) based at least partially on the speech rate of a particular speaker, according to several embodiments of the present disclosure. In various embodiments, the EOS threshold is determined using method 500, as described above. In embodiments, method 600 is implemented by an audio capture device, as described in the present disclosure. In yet other embodiments, method 600 is implemented as a component of a speech processing service implemented by a computing resource service provider. For example, method 600 is implemented by an EOS detector (e.g., an EOS detector 210, as described in more detail above) running as a component of a speech processing pipeline implemented by a computing resource service provider using a cloud computing device.

[0041] Method 600 includes obtaining a character string in block 602. In various embodiments, the Connectionist Temporal Classification (CTC) function generates a probability distribution of characters as an output, which is at least partially based on the output of the acoustic model. For example, as described above, the CTC function considers the input features of the audio signal at each time step and outputs a probability distribution for the possible characters at each time step. Furthermore, as described above, a component of the speech recognition pipeline (e.g., a greedy decoder) may use an algorithm to generate a character string based at least partially on the probability distribution output by the CTC.

[0042] In block 604, the system implementing method 600 evaluates the first sliding window. As described above, the sliding window may contain multiple time steps or audio frames. In one example, a 500-millisecond sliding window contains 25 time steps, each time step having a duration of 20 milliseconds. Various different lengths for the sliding window may be used in connection with method 600. Furthermore, in embodiments, the sliding window corresponds to a set of characters in a string of characters. For example, as described above in relation to Figure 3, the sliding window 352 contains a substring of characters from a string of characters. In various embodiments, the sliding window begins at time 0 (e.g., the first time step) and contains a continuous set of time steps or audio frames (e.g., from time 0 to time X).

[0043] In step 606, the system implementing method 600 determines the proportion of blank symbols in the first sliding window. In one embodiment, the proportion of blank symbols is determined by dividing the number of blank symbols in the sliding window by the total number of symbols in the sliding window. In another embodiment, as an example where the sliding window size does not change, the proportion of blank symbols is determined by dividing the number of blank symbols by the length of the sliding window (e.g., the number of time steps or audio frames).

[0044] In block 608, the system implementing method 600 determines whether the EOS threshold has been met, at least in part, based on the percentage of blank symbols determined in block 606. For example, if the percentage of blank symbols in the first sliding window is 95% and the EOS threshold is 90%, the system implementing method 600 proceeds to block 610. In block 610, the system implementing method 600 flags the sliding window as EOS. As described above, flagging a particular sliding window as EOS can trigger components of the speech processing pipeline to generate a transcript for use by another application and / or device. Still returning to block 608, if the system implementing method 600 determines that the EOS threshold has not been met, method 600 returns to block 604 and proceeds to the next sliding window. In embodiments, the system implementing method 600 increments the sliding window by a single time step or audio frame. For example, if the current sliding window is from time step 0 to time step 25, the sliding window is modified to cover the range from time step 1 to time step 26. In various embodiments, the number of time steps or audio frames over which the sliding window is moved may also be modified.

[0045] Figure 7 shows a parallel processing unit ("PPU") 700 according to one embodiment. In the embodiment, the PPU 700 consists of machine-readable code that, when executed by the PPU, causes the PPU to perform some or all of the processes and techniques described throughout this disclosure. In the embodiment, the PPU 700 is a multithreaded processor implemented on one or more integrated circuit devices and utilizing multithreading as a latency-hiding technique designed to process computer-readable instructions (also called machine-readable instructions or simply instructions) in parallel on multiple threads. In the embodiment, a thread refers to a thread of execution and is an embodiment of a set of instructions configured to be executed by the PPU 700. In the embodiment, the PPU 700 is an image processing unit ("GPU") configured to implement a graphics rendering pipeline for processing three-dimensional ("3D") graphics data to generate two-dimensional ("2D") image data for display on a display device such as a liquid crystal display (LCD) device. In the embodiment, the PPU700 is used to perform calculations such as linear algebra and machine learning operations. Figure 7 shows an example parallel processor for illustrative purposes only, but it should be interpreted that any suitable processor may be employed as a non-limiting example of the processor architecture assumed within the scope of this disclosure, and as a supplement and / or substitute for the same.

[0046] In the embodiments, one or more PPUs are configured to accelerate high-performance computing ("HPC"), data center, and machine learning applications. In the embodiments, the PPU700 is configured to accelerate deep learning systems and applications, including, but not limited to, autonomous vehicle platforms, deep learning, high-precision speech, image and text recognition systems, intelligent video analysis, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations.

[0047] In an embodiment, the PPU 700 includes an input / output ("I / O") unit 706, a front-end unit 710, a scheduler unit 712, a work distribution unit 714, a hub 716, a crossbar ("Xbar") 720, one or more integrated processing clusters ("GPC") 718, and one or more partition units 722. In an embodiment, the PPU 700 is connected to a host processor or other PPU 700 via one or more high-speed GPU interconnects 708. In an embodiment, the PPU 700 is connected to a host processor or other peripheral devices via an interconnect 702. In an embodiment, the PPU 700 is connected to local memory including one or more memory devices 704. In an embodiment, the local memory comprises one or more dynamic random access memory ("DRAM") devices. In the embodiment, one or more DRAM devices are configured and / or configurable as a high-bandwidth memory ("HBM") subsystem in which multiple DRAM dies are stacked within each device.

[0048] The High-Speed ​​GPU Interconnect 708 may refer to a wire-based multi-lane communication link used by the system to scale and include one or more PPUs 700 coupled with one or more CPUs, and supporting cache coherence and CPU mastering between the PPUs 700 and the CPUs. In the embodiment, data and / or commands are transmitted by the High-Speed ​​GPU Interconnect 708 through the hub 716 to other units of the PPU 700, such as one or more copy engines, video encoders, video decoders, power management units, and other components, which may not be explicitly shown in Figure 7.

[0049] In the embodiment, the I / O unit 706 is configured to transmit and receive communications (e.g., commands, data) from a host processor (not shown in Figure 7) over the system bus 702. In the embodiment, the I / O unit 706 communicates with the host processor directly via the system bus 702 or through one or more intermediate devices such as a memory bridge. In the embodiment, the I / O unit 706 may communicate with one or more other processors, such as one or more PPUs 700, via the system bus 702. In the embodiment, the I / O unit 706 implements a Peripheral Component Interconnect Express ("PCIe") interface for communication over the PCIe bus. In the embodiment, the I / O unit 706 implements an interface for communicating with external devices.

[0050] In the embodiment, the I / O unit 706 decodes packets received via the system bus 702. In the embodiment, at least some packets represent commands configured to cause the PPU 700 to perform various operations. In the embodiment, the I / O unit 706 transmits the decoded commands to various other units of the PPU 700 as specified by the commands. In the embodiment, the commands are transmitted to the front-end unit 710 and / or to the hub 716, or to one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown in Figure 7). In the embodiment, the I / O unit 706 is configured to route communication between and within the various logical units of the PPU 700.

[0051] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 700 for processing. In one embodiment, the workload includes instructions and data that will be processed by these instructions. In one embodiment, the buffer is a region in memory accessible (e.g., read / write) by both the host processor and the PPU 700, and the host interface unit may be configured to access the buffer in system memory connected to the system bus 702 via memory requests transmitted over the system bus 702 by the I / O unit 706. In one embodiment, the host processor writes the command stream to the buffer and then transmits a pointer to the start of the command stream to the PPU 700 so that the front-end unit 710 receives a pointer to one or more command streams, manages one or more streams, reads commands from the streams, and forwards the commands to various units of the PPU 700.

[0052] In the embodiment, the front-end unit 710 is coupled to a scheduler unit 712 which configures various GPCs 718 to handle tasks defined by one or more streams. In the embodiment, the scheduler unit 712 is configured to track status information about the various tasks managed by the scheduler unit 712, where the status information may indicate which GPC 718 a task is assigned to, whether the task is active or inactive, the priority level associated with the task, etc. In the embodiment, the scheduler unit 712 manages the execution of multiple tasks on one or more GPCs 718.

[0053] In one embodiment, the scheduler unit 712 is coupled to a work distribution unit 714 configured to dispatch tasks for execution on the GPC 718. In one embodiment, the work distribution unit 714 tracks several scheduled tasks received from the scheduler unit 712, and the work distribution unit 714 manages a waiting task pool and an active task pool for each of the GPCs 718. In one embodiment, the waiting task pool may have several slots (e.g., 32 slots) for tasks assigned to be processed by a particular GPC 718, and the active task pool may have several slots (e.g., 4 slots) for tasks being actively processed by the GPC 718 such that when the GPC 718 completes the execution of a task, that task is removed from the active task pool for the GPC 718, and one of the other tasks from the waiting task pool is selected and scheduled for execution on the GPC 718. In the embodiment, if an active task is inactive on the GPC718, such as while waiting for data dependencies to be resolved, the active task is removed from the GPC718 and returned to the waiting task pool, while another task in the waiting task pool is selected and scheduled for execution on the GPC718.

[0054] In this embodiment, the work distribution unit 714 communicates with one or more GPCs 718 via the XBar 720. In this embodiment, the XBar 720 is an interconnection network that can be configured to connect many units of the PPU 700 to other units of the PPU 700 and to connect the work distribution unit 714 to a specific GPC 718. Although not explicitly shown, one or more other units of the PPU 700 may also be connected to the XBar 720 via the hub 716.

[0055] Tasks are managed by a scheduler unit 712 and dispatched to a GPC 718 by a work distribution unit 714. The GPC 718 is configured to process tasks and produce results. The results can be used by other tasks within the GPC 718, routed to a different GPC 718 via the XBar 720, or stored in memory 704. The results can be written to memory 704 via a partition unit 722, which implements a memory interface for reading and writing data to and from memory 704. The results can be transmitted to another PPU 700 or CPU via a high-speed GPU interconnect 708. In this embodiment, the PPU 700 includes several U partition units 722, equal to the number of separate and distinct memory devices 704 connected to the PPU 700. The partition units 722 are described in more detail below, with reference to Figure 9.

[0056] In an embodiment, the host processor runs a driver kernel that implements an Application Programming Interface ("API"), enabling one or more applications running on the host processor to schedule operations for execution on the PPU700. In an embodiment, multiple computing applications run concurrently on the PPU700, and the PPU700 provides isolation, quality of service ("QoS"), and independent address spaces to the multiple computing applications. In an embodiment, the application generates instructions (e.g., in the form of API calls) that cause the driver kernel to generate one or more tasks for execution on the PPU700, and the driver kernel outputs the tasks to one or more streams being processed by the PPU700. In an embodiment, each task includes one or more groups of related threads, which may be called a warp. In an embodiment, a warp includes multiple related threads (e.g., 32 threads) that can run in parallel. In an embodiment, a collaborative thread may refer to multiple threads that include instructions for task execution and exchange data through shared memory. With Figure 10, a thread and a cooperating thread in one embodiment are described in more detail.

[0057] Figure 8 shows a GPC800, such as the GPC shown in Figure 7, of the PPU700, according to one embodiment. In the embodiment, each GPC800 includes several hardware units for processing tasks, and each GPC800 includes a pipeline manager 802, a pre-raster operations unit ("PROP") 804, a raster engine 808, a work distribution crossbar ("WDX") 816, a memory management unit ("MMU") 818, one or more data processing clusters ("DPC") 806, and any appropriate combination of components. It will be understood that the GPC800 in Figure 8 may include other hardware units in place of or in addition to the units shown in Figure 8.

[0058] In the embodiment, the operation of the GPC800 is controlled by the pipeline manager 802. The pipeline manager 802 manages the configuration of one or more DPC806s for handling tasks allocated to the GPC800. In the embodiment, the pipeline manager 802 configures at least one of the one or more DPC806s to implement at least a portion of the graphics rendering pipeline. In the embodiment, the DPC806 is configured to run a vertex shader program on a programmable streaming multiprocessor ("SM": streaming multiprocessor) 814. The pipeline manager 802 is configured to route packets received from the work distribution to the appropriate logical unit in the GPC800, with some packets being routed to fixed-function hardware units in PROP804 and / or raster engine 808, while other packets are routed to the DPC806 for processing by the primitive engine 812 or SM814. In this embodiment, the pipeline manager 802 configures at least one of one or more DPC 806 to implement a neural network model and / or computing pipeline.

[0059] In the embodiment, the PROP unit 804 is configured to route the data generated by the raster engine 808 and DPC 806 to the raster operation ("ROP") unit in the memory partition unit described in more detail above. In the embodiment, the PROP unit 804 is configured to perform optimizations such as color fusion, pixel data organization, and address translation. The raster engine 808 includes several fixed-function hardware units configured to perform various raster operations in the embodiment, and the raster engine 808 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, a tile coalescing engine, and any appropriate combination thereof. In an embodiment, the setup engine receives the transformed vertices and generates a plane equation associated with the geometric primitives defined by the vertices. The plane equation is transmitted to the coarse raster engine to generate coverage information about the primitives (e.g., x, y coverage masks for tiles). The output of the coarse raster engine is transmitted to the culling engine, where fragments associated with primitives that fail the z test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. In an embodiment, fragments that survive clipping and culling are transmitted to the fine raster engine to generate pixel fragment attributes based on the plane equation generated by the setup engine. In an embodiment, the output of the raster engine 808 contains fragments that will be processed by any appropriate entity, such as a fragment shader implemented within the DPC806.

[0060] In the embodiment, each DPC806 included in the GPC800 comprises an M-Pipe Controller ("MPC") 810, a primitive engine 812, one or more SM814s, and any suitable combination thereof. In the embodiment, the MPC810 controls the operation of the DPC806 and routes packets received from the pipeline manager 802 to the appropriate unit within the DPC806. In the embodiment, packets associated with vertices are routed to the primitive engine 812, which is configured to fetch vertex attributes associated with the vertex from memory, while packets associated with shader programs may be transmitted to the SM814.

[0061] In one embodiment, the SM814 comprises a programmable streaming processor configured to handle tasks represented by several threads. In another embodiment, the SM814 is multithreaded and configured to execute multiple threads (e.g., 32 threads) from a particular group of threads simultaneously, implementing a SIMD (Single Instruction, Multiple Data) architecture where each thread within the group of threads (e.g., a warp) is configured to process a different set of data based on the same set of instructions. In yet another embodiment, all threads within the group of threads execute the same instruction. In yet another embodiment, the SM814 implements a SIMT (Single Instruction, Multiple Thread) architecture where each thread within the group of threads is configured to process a different set of data based on the same set of instructions, but individual threads within the group of threads are allowed to branch during execution. In yet another embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and continuous execution within warps when threads within warps branch. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, enabling equal concurrency across all threads, within warps, and between warps. In the embodiment, the execution state is maintained for each individual thread and for threads executing the same instruction that may be concentrated and executed in parallel for better efficiency. In the embodiment, the SM814 is described in more detail below.

[0062] In the embodiment, the MMU818 provides an interface between the GPC800 and the memory partition unit, and the MMU818 provides translation of virtual addresses to physical addresses, memory protection, and arbitration of memory requests. In the embodiment, the MMU818 provides one or more translation lookaside buffers ("TLBs") for performing the translation of virtual addresses to physical addresses in memory.

[0063] Figure 9 shows a memory partition unit of a PPU according to one embodiment. In the embodiment, the memory partition unit 900 includes a raster operation ("ROP") unit 902, a level 2 ("L2") cache 904, a memory interface 906, and any suitable combination thereof. The memory interface 906 is coupled to memory. The memory interface 906 may implement a 32, 64, 128, 1024-bit data bus or similar for high-speed data transfer. In the embodiment, the PPU incorporates U memory interfaces 906, one for each pair of partition units 900, and each pair of partition units 900 is coupled to a corresponding memory device. For example, the PPU may be connected to up to Y memory devices, such as a high-bandwidth memory stack or graphics double-data-rate, version 5, synchronous dynamic random access memory ("GDDR5 SDRAM").

[0064] In this embodiment, the memory interface 906 implements an HBM2 memory interface, where Y is equal to half of U. In this embodiment, the HBM2 memory stack is located on the same physical package as the PPU, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In this embodiment, each HBM2 stack includes four memory dies, where Y is equal to 4, and the HBM2 stack includes a total of eight channels and two 128-bit channels per die with a 1024-bit data bus width.

[0065] In this embodiment, the memory supports Single-Error Correcting Double-Error Detecting ("SECDED") Error Correction Code ("ECC") to protect data. ECC provides greater reliability for computational applications that are susceptible to data corruption. Reliability is particularly important in large-scale cluster computing environments where PPUs process very large datasets and / or run applications for extended periods.

[0066] In the embodiment, the PPU implements a multi-level memory hierarchy. In the embodiment, the memory partition unit 900 supports integrated memory to provide a single unified virtual address space for CPU and PPU memory, enabling data sharing between virtual memory systems. In the embodiment, the frequency of PPU access to memory located on other processors is traced to ensure that memory pages are moved to the physical memory of the PPU that accesses the pages more frequently. In the embodiment, the high-speed GPU interconnect 708 supports address translation services that enable the PPU to directly access the CPU's page table, providing the PPU with full access to CPU memory.

[0067] In the embodiment, the copy engine transfers data between multiple PPUs or between a PPU and a CPU. In the embodiment, the copy engine can generate a page fault for an address not mapped to a page table, the memory partition unit 900 then services the page fault and maps the address to a page table, after which the copy engine performs the transfer. In the embodiment, memory is pinned (i.e., non-paging) for multiple copy engine operations between multiple processors, significantly reducing the available memory. In the embodiment, when hardware generates a page fault, the address can be passed to the copy engine regardless of whether the memory page is resident or not, making the copy process straightforward.

[0068] Data from the memory shown in Figure 7 or other system memory is fetched by the memory partition unit 900 and stored in the L2 cache 904, which is located on the chip and shared among various GPCs in one embodiment. Each memory partition unit 900 in the embodiment includes at least a portion of the L2 cache 904 associated with the corresponding memory device. In the embodiment, lower levels of cache are implemented in various units within the GPC. In the embodiment, each SM940 may implement a level 1 ("L1") cache, which is private memory dedicated to a particular SM, and data from the L2 cache 904 is fetched and stored in each of the L1 caches for processing in the functional units of the SM. In the embodiment, the L2 cache 904 is coupled to the memory interface 906 and the XBar720.

[0069] ROP unit 902 performs graphics raster operations related to pixel color, such as color compression and pixel merging, in the embodiment. ROP unit 902 implements depth testing together with the raster engine and receives the depth of sample locations associated with pixel fragments from the raster engine's culling engine. In the embodiment, the depth is tested against the corresponding depth in the depth buffer of the sample locations associated with the fragment. In the embodiment, if the fragment passes the depth test of the sample locations, ROP unit 902 updates the depth buffer and transmits the result of the depth test to the raster engine. The number of partition units 900 can be different from the number of GPCs, and it will be understood that each ROP unit 902 can be concatenated to each of the GPCs in the embodiment. In the embodiment, ROP unit 902 tracks packets received from different GPCs and determines which of the results generated by ROP unit 902 are routed to Xbar.

[0070] Figure 10 shows a streaming multiprocessor, such as the streaming multiprocessor in Figure 8, according to one embodiment. In the embodiment, SM1000 includes an instruction cache 1002, one or more scheduler units 1004, a register file 1008, one or more processing cores 1010, one or more special function units ("SFUs") 1012, one or more load / store units ("LSUs") 1014, an interconnect network 1016, a shared memory / L1 cache 1018, and any appropriate combination thereof. In the embodiment, a work distribution unit dispatches tasks for execution on the GPC of the PPU, each task is allocated to a specific DPC within the GPC, and if the task is associated with a shader program, the task is allocated to SM1000. In one embodiment, the scheduler unit 1004 receives tasks from the work distribution unit and manages instruction scheduling for one or more thread blocks assigned to the SM1000. In one embodiment, the scheduler unit 1004 schedules thread blocks for execution as warps of parallel threads, and each thread block is allocated at least one warp. In one embodiment, each warp executes a thread. In one embodiment, the scheduler unit 1004 manages multiple different thread blocks, allocates warps to different thread blocks, and then, during each clock cycle, dispatches instructions from multiple different cooperation groups to various functional units (e.g., core 1010, SFU 1012, and LSU 1014).

[0071] A cooperating group can refer to a programming model for organizing groups of communicating threads, allowing developers to represent the granularity at which threads are communicating, enabling a richer and more efficient representation of parallel decomposition. In an example, the cooperating invocation API supports synchronization between thread blocks for the execution of parallel algorithms. In an example, an application of the traditional programming model provides a single, simple structure for synchronizing cooperating threads, such as a barrier across all threads in a thread block (e.g., the syncthreads() function). Nevertheless, programmers often want to define groups of threads smaller than the thread block granularity, and synchronize within groups defined to allow for greater performance, design flexibility, and software reuse in the form of a functional interface for the entire collective group. Cooperating groups allow programmers to explicitly define groups of threads at subblock (i.e., as small as a single thread) and multi-block granularity, and to perform collective actions, such as synchronization of threads within a cooperating group. The programming model supports clean constructs that span software boundaries, allowing libraries and utility functions to synchronize reliably within their local contexts without the need to make assumptions about centralization. The cooperative group primitive enables new patterns of cooperative parallelism, including producer-consumer parallelism, opportunityistic parallelism, and global synchronization across the entire grid of thread blocks.

[0072] In the embodiment, the dispatch unit 1006 is configured to transmit instructions to one or more functional units, and the scheduler unit 1004 includes two dispatch units 1006 that enable two different instructions from the same warp to be dispatched during each clock cycle. In the embodiment, each scheduler unit 1004 includes a single dispatch unit 1006 or an additional dispatch unit 1006.

[0073] Each SM1000 includes, in an embodiment, a register file 1008 that provides a set of registers for the functional units of the SM1000. In an embodiment, the register file 1008 is divided among the functional units such that each functional unit is allocated a dedicated portion of the register file 1008. In an embodiment, the register file 1008 is divided among different warps being executed by the SM1000, and the register file 1008 provides temporary storage for operands connected to the data paths of the functional units. In an embodiment, each SM1000 comprises a plurality of L processing cores 1010. In an embodiment, the SM1000 includes many (e.g., 128 or more) distinct processing cores 1010. Each core 1010 includes, in an embodiment, a fully pipelining single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point logic unit and an integer arithmetic logic unit. In this embodiment, the floating-point logic unit implements the IEEE 754-2008 standard for floating-point operations. In this embodiment, core 1010 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0074] The tensor core is configured to perform matrix operations according to the embodiment. In the embodiment, one or more tensor cores are included in core 1010. In the embodiment, the tensor core is configured to perform deep learning matrix operations, such as convolution operations for training and inference of neural networks. In the embodiment, each tensor core operates on a 4x4 matrix and performs matrix multiplication and cumulative operations D = A x B + C, where A, B, C, and D are 4x4 matrices.

[0075] In the embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, and accumulation matrices C and D are either 16-bit floating-point or 32-bit floating-point matrices. In the embodiment, TensorCore performs operations on 16-bit floating-point input data with 32-bit floating-point accumulation. In the embodiment, 16-bit floating-point multiplication requires 64 operations, resulting in a full-precision product that is then accumulated using 32-bit floating-point addition with other intermediate products for 4x4x4 matrix multiplication. In the embodiment, TensorCore is used to perform much larger two-dimensional or greater matrix operations constructed from these smaller elements. In the embodiment, APIs such as the CUDA9 C++ API exhibit specialized matrix loading, matrix multiplication and accumulation, as well as matrix store operations, to efficiently use TensorCore from CUDA-C++ programs. In the embodiment, at the CUDA level, the warp-level interface assumes a 16x16 size matrix spanning all 32 threads of the warp.

[0076] In the embodiment, each SM1000 comprises M SFU1012 that perform special functions (e.g., attribute evaluation, reciprocal square root, and similar). In the embodiment, the SFU1012 includes a tree traverse unit configured to traverse a hierarchical tree data structure. In the embodiment, the SFU1012 includes a texture unit configured to perform texture map filtering operations. In the embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory, sample the texture map, and produce sampled texture values ​​for use in a shader program executed by the SM1000. In the embodiment, the texture map is stored in a shared memory / L1 cache. The texture unit implements texture operations, such as filtering operations using mipmaps (e.g., texture maps with varying levels of detail), as in one embodiment. In the embodiment, each SM1000 comprises two texture units.

[0077] Each SM1000 comprises N LSU954s that implement load and store operations between the shared memory / L1 cache 1018 and the register file 1008, as in the embodiment. Each SM1000 includes an interconnect network 1016 that connects each of the functional units to the register file 1008, and connects the LSU1014 to the register file 1008 and the shared memory / L1 cache 1018, as in the embodiment. In the embodiment, the interconnect network 1016 is a crossbar that can be configured to connect any of the functional units to any of the registers in the register file 1008, and connect the LSU1014 to the register file and storage locations in the shared memory / L1 cache 1018.

[0078] The shared memory / L1 cache 1018 is an array of on-chip memory that enables data storage and communication between the SM1000 and the primitive engine, and between threads in the SM1000, in the embodiment. In the embodiment, the shared memory / L1 cache 1018 has a storage capacity of 128KB and is located in the path from the SM1000 to the partition unit. In the embodiment, the shared memory / L1 cache 1018 is used to cache reads and writes. One or more of the shared memory / L1 cache 1018, the L2 cache, and the memory are auxiliary storage devices.

[0079] Combining data cache and shared memory functions into a single memory block provides improved performance for both types of memory access in embodiments. The capacity is such that, in embodiments, if shared memory is configured to use half of the capacity, texture and load / store operations can use the remaining capacity, or the capacity can be used or made available for use as a cache by programs that do not use shared memory. The integration within the shared memory / L1 cache 1018 allows the shared memory / L1 cache 1018 to function as a high-throughput tube for streaming data, while simultaneously providing high-bandwidth and low-latency access to frequently reused data, as in embodiments. When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics processing. In embodiments, a fixed-function image processing unit is bypassed, creating a much simpler programming model. In a general-purpose parallel computing configuration, the work distribution unit, in embodiments, directly allocates and distributes blocks of threads to the DPC. The threads within the block execute the same program, using a unique thread ID at computation time to ensure that each thread produces a unique result, using the SM1000 to execute the program and perform the computation, using the shared memory / L1 cache 1018 to communicate between threads, and using the LSU1014 to read and write global memory through the shared memory / L1 cache 1018 and the memory partition unit, as in one embodiment. In the embodiment, when configured for general-purpose parallel computing, the SM1000 writes commands that the scheduler unit can use to start new work on the DPC.

[0080] In embodiments, the PPU is included in or linked to desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants ("PDAs"), digital cameras, vehicles, head-mounted displays, handheld electronic devices, etc. In embodiments, the PPU is embodied on a single semiconductor substrate. In embodiments, the PPU is included in a system-on-a-chip ("SoC") together with one or more other devices such as additional PPUs, memory, a reduced instruction set computer ("RISC") CPU, a memory management unit ("MMU"), a digital-to-analog converter ("DAC"), and the like.

[0081] In some embodiments, the PPU may be contained on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on the motherboard of a desktop computer. In yet another embodiment, the PPU may be an integrated graphics processing unit ("iGPU") included in the chipset of the motherboard.

[0082] Figure 11 shows a computer system 1100 in one embodiment, which can implement various architectures and / or functions. In embodiments, computer system 1100 is configured to implement various processes and methods described throughout this disclosure. In one or more embodiments, computer system 1100 may be one of several similarly configured computer systems that operate together within or as a component of a data center and are used for tasks such as training machine learning models, providing a cloud computing environment, hosting one or more virtual machines, and other high-performance computing tasks. In one or more embodiments, computer system 1100 may be an edge device such as an IoT device that is communicably connected to one or more other Internet of Things (IoT) devices and / or on-site servers.

[0083] In the embodiment, the computer system 1100 includes at least one central processing unit 1102 connected to a communication bus 1110, which is implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol. In the embodiment, the computer system 1100 includes main memory 1104 and control logic (implemented, for example, as hardware, software, or a combination thereof), and data is stored in the main memory 1104, which may take the form of random access memory ("RAM"). In the embodiment, a network interface subsystem 1122 provides interfaces to other computing devices and networks for receiving data from the computer system 1100 and transmitting data to other systems.

[0084] The computer system 1100, in an embodiment, includes an input device 1108, a parallel processing system 1112, and a display device 1106 (which can be implemented using a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, or other suitable display technology). In an embodiment, user input is received from the input device 1108, such as a keyboard, mouse, touchpad, or microphone. In an embodiment, each of the aforementioned modules can be placed on a single semiconductor platform to form a processing system.

[0085] In this explanation, a single semiconductor platform may refer to a single, integrated semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" also refers to multi-chip modules with improved connectivity that simulate on-chip operation, which can offer significant improvements over traditional central processing units ("CPUs") and bus implementation configurations. Naturally, various modules can be placed separately or in various combinations of semiconductor platforms, according to user requirements.

[0086] In the embodiment, the computer program is stored in the main memory 1104 and / or secondary storage in the form of machine-readable executable code or computer-controlled logic algorithms. When the computer program is executed by one or more processors, it enables the system 1100 to perform various functions according to one embodiment. Memory 1104, storage, and / or any other storage are possible examples of computer-readable media. Secondary storage may refer to any suitable storage device or system, such as a hard disk drive and / or a removable storage drive (representing a floppy disk drive, magnetic tape drive, compact disk drive, digital versatile disk ("DVD") drive, recording device, or universal serial bus ("USB") flash memory).

[0087] In the embodiments, the architectures and / or functions of the various figures above are implemented in the context of a central processing unit 1102, a parallel processing system 1112, an integrated circuit having at least some of the capabilities of both the central processing unit 1102 and the parallel processing system 1112, a chipset (e.g., a group of integrated circuits designed and sold to function as a single unit for performing the relevant functions), and any suitable combination of integrated circuits.

[0088] In embodiments, the architectures and / or functions of the various figures above are implemented in the context of general computer systems, circuit board systems, game console systems for entertainment purposes, application-specific systems, and so on. In embodiments, the computer system 1100 may take the form of a desktop computer, laptop computer, tablet computer, server, supercomputer, smartphone (e.g., wireless, handheld device), personal digital assistant ("PDA"), digital camera, vehicle, head-mounted display, handheld electronic device, mobile phone device, television, workstation, game console, embedded system, and / or any other type of logic.

[0089] In the embodiment, the parallel processing system 1112 includes a plurality of PPUs 1114 and associated memory 1116. In the embodiment, the PPUs are connected to a host processor or other peripheral devices via an interconnect 1118 and a switch 1120 or multiplexer. In the embodiment, the parallel processing system 1112 distributes computation tasks across parallelizable PPUs 1114 (e.g., as part of distributing computation tasks across a plurality of GPU thread blocks). In the embodiment, memory is shared and accessible across some or all of the PPUs 1114 (e.g., for read and / or write access), but such shared memory may suffer performance disadvantages compared to the use of local memory and registers resident in the PPUs. In the embodiment, the operation of the PPUs 1114 is synchronized through the use of commands such as syncthreads(), which requests all threads in a block (e.g., running across a plurality of PPUs 1114) to reach a specific point in the execution of the code before proceeding.

[0090] This specification and its drawings should therefore be evaluated in an illustrative rather than restrictive sense. Nevertheless, it will be apparent that various modifications and changes can be made to this specification and its drawings without departing from the broader spirit and scope of the invention, as set forth in the claims.

[0091] Other variations are within the spirit of this disclosure. Therefore, the techniques of the disclosure are susceptible to various modifications and alternative structures, though their specific examples have been shown in the drawings and described in detail above. Nevertheless, it should be understood that there is no intention to limit the invention to any particular form or form of disclosure, but rather to cover all modifications, alternative structures, and equivalents that fall within the spirit and scope of the invention, as defined in the appended claims.

[0092] The use of the terms “a,” “an,” and “the” and similar referential concepts in the context describing embodiments of the disclosure (particularly in the context of the following claims) should be interpreted as covering both singular and plural forms unless otherwise indicated herein or unless explicitly contradicted by the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as unrestricted terms (i.e., “including but not limited to”) unless otherwise stated. The term “connected,” when unmodified and referring to a physical connection, should be interpreted as being partially or fully housed, attached, or joined together, even if there are intervening elements. The enumeration of value ranges herein is intended merely as a concise way of referring individually to each separate value included in the range unless otherwise indicated herein, and each separate value is incorporated herein, even if individually enumerated herein. The use of the terms “set” (e.g., “set of items”) or “subset” should be interpreted as a non-empty collection containing one or more members, unless otherwise stated or inconsistent with the context. Furthermore, unless otherwise stated or inconsistent with the context, the term “subset” of a corresponding set does not necessarily refer to a canonical subset of the corresponding set, and a subset and a corresponding set can be equivalent.

[0093] Conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are generally understood in context to indicate that an item, term, etc., is A or B or C, or any non-empty subset of the set A, B, and C, unless otherwise specifically determined or unless otherwise clearly contradicted by the context. For example, in an example illustrating a set with three members, the conjunctions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the sets {A}, {B}, {C}, {A,B}, {A,C}, {B,C}, or {A,B,C}. Thus, such conjunctions are not generally intended to suggest that a particular embodiment must contain at least one of A, at least one of B, and at least one of C, respectively. Furthermore, unless otherwise stated or unless inconsistent with the context, the term “plural” refers to the state of being multiple (for example, “multiple items” refers to multiple items). The number of items in the plural is at least two, but can be more when explicitly or as indicated by the context. Furthermore, unless otherwise determined or otherwise evident from the context, the phrase “based on” means “at least partially based on” and does not mean “simply based on”

[0094] The operations of the processes described herein can be performed in any suitable order, unless otherwise specified herein or unless otherwise explicitly contradicted by the context. In embodiments, these processes (or variations and / or combinations thereof) described herein are performed under the control of one or more computer systems consisting of executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are executed together on one or more processors by hardware or a combination thereof. In embodiments, the code is stored on a computer-readable storage medium, for example, in the form of a computer program containing multiple instructions executable by one or more processors. In embodiments, the computer-readable storage medium is a non-temporary computer-readable storage medium excluding temporary signals (e.g., transient electrical or electromagnetic transmissions that propagate), but includes non-temporary data storage circuit equipment (e.g., buffers, caches, and queues) for transmitting and receiving temporary signals. In embodiments, code (e.g., executable code or source code) is stored in one or more sets of non-temporary computer-readable storage media (or other memory for storing executable instructions) that, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. The set of non-temporary computer-readable storage media, in embodiments, includes a plurality of non-temporary computer-readable storage media and one or more individual non-temporary storage media of the plurality of non-temporary computer-readable storage media, where the plurality of non-temporary computer-readable storage media collectively store all of the code, but which do not contain all of the code. In embodiments, the executable instructions are executed such that different instructions are executed by different processors (e.g., a non-temporary computer-readable storage medium stores instructions, and the main CPU executes some of the instructions while the graphics processor unit executes other instructions).In this embodiment, different components of the computer system have separate processors, and each processor executes a different subset of instructions.

[0095] Accordingly, in embodiments, a computer system is configured to implement one or more services that perform the operations of the processing described herein, either individually or collectively, and such a computer system consists of applicable hardware and / or software that enables the performance of the operations. Furthermore, a computer system implementing an embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system is a distributed computer system comprising multiple devices, each operating in such a way that the distributed computer system performs the operations described herein, and the single device does not perform all of the operations.

[0096] The use of any and all examples or illustrative words (e.g., “etc.”) provided herein is solely intended to better illustrate embodiments of the invention and, unless otherwise claimed, does not limit the scope of the invention. Words herein should not be construed as referring to any element not claimed as essential to the practice of the invention.

[0097] Examples of the present disclosure, including the best mode known to the inventors for carrying out the present invention, are described herein. Variations of these examples may become apparent to those skilled in the art by reading the preceding description. The inventors expect that those skilled in the art will adopt such variations as needed, and the inventors intend that the examples of the present disclosure may be practiced in ways other than those specifically described herein. Accordingly, the scope of the present disclosure includes all variations and equivalents of the subject matter enumerated in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the elements described above in all possible variations is covered by the scope of the present disclosure unless otherwise indicated herein or unless otherwise clearly contradicted in context.

[0098] All references cited herein, including publications, patent applications, and patents, are incorporated by reference herein to the same extent as they are individually and specifically indicated and as a whole.

[0099] In this description and claims, the terms “connected” and “linked” may be used together with their derivatives. It should be understood that these terms are not intended to be synonyms of each other. Rather, in certain examples, “connected” or “linked” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. “Linked” may also mean that two or more elements are not in direct contact with one another but still cooperate or interact with one another.

[0100] Unless otherwise specifically defined, throughout this Specification, terms such as “processing,” “computing,” “calculating,” “determining,” or similar shall be understood to refer to the actions and / or processing of a computer or computing system or similar electronic computing device that manipulates and / or converts data represented as electronic or other physical quantities in the registers and / or memory of a computing system to other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of a computing system.

[0101] In a similar manner, the term “processor” may refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. In non-limiting examples, “processor” may be a central processing unit (CPU) or an image processing unit (GPU). A “computing platform” may comprise one or more processors. As used herein, “software” processing may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Each processing may also refer to multiple processing operations for executing instructions sequentially or in parallel, continuously or intermittently. The terms “system” and “method” are used herein without distinction to the extent that a system can embody one or more methods and a method can be considered a system.

[0102] This document may refer to acquiring, obtaining, receiving, or inputting analog or digital data into subsystems, computer systems, or computer-implemented machines. The processing of acquiring, obtaining, receiving, or inputting analog and digital data can be achieved in various ways, such as by receiving data as a function call to an application programming interface or as a parameter to a call. In some implementations, the processing of acquiring, obtaining, receiving, or inputting analog or digital data can be achieved by transferring data via a serial or parallel interface. In other implementations, the processing of acquiring, obtaining, receiving, or inputting analog or digital data can be achieved by transferring data via a computer network from a providing entity to a receiving entity. References may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the processing of providing, outputting, transmitting, sending, or presenting analog or digital data can be achieved by transferring data as an input or output parameter to a function call, a parameter to an application programming interface, or as an inter-process communication mechanism.

[0103] While the above discussion illustrates an example implementation of the techniques described, other architectures may be used to implement the described functions and are intended to be within the scope of this disclosure. Furthermore, while a specific distribution of responsibility is defined above for the purposes of the discussion, various functions and responsibilities may be distributed and divided in different ways depending on the context.

[0104] At least one embodiment of the present disclosure can be described in consideration of the following sections. 1. Determining the speech end (EOS) threshold based at least partially on speech speed, For at least one segment of the set of segments in the audio input, predict the set of characters that represent at least one segment, Determining the EOS for at least one segment based at least in part on the determination that the EOS threshold is met for the predicted set of characters for at least one segment. Methods that include... 2. The method of Section 1, wherein the predicted set of characters for a segment is generated by a Connectionist Temporal Classification (CTC) function that generates the probability distribution of one character in the set of characters as an output. 3. The method of Section 2, in which the speech rate is determined at least in part on a set of word intervals. 4. The method of Section 3, wherein determining the EOS threshold further includes modifying the EOS threshold based at least in part on the maximum interword interval in the set of interword intervals. 5. The method of Section 3 or 4, wherein determining the EOS threshold further includes modifying the EOS threshold based at least in part on the median value of a set of interword intervals. 6. Any method from Sections 3–5, wherein determining the EOS threshold further includes modifying the EOS threshold based at least in part on the variance of the set of interword intervals and the maximum interword interval of the set of interword intervals. 7. Any method from Sections 3-6, wherein the set of word intervals is calculated by determining at least the time between a first subset of characters in a set of characters and a second subset of characters in a set of characters. 8. The method of Section 7, where time corresponds to some segments of a set of segments. 9. The method of Section 7 or 8, wherein the set of characters is determined by applying at least a greedy selection algorithm to the output of the CTC function. 10. The method of Section 9, wherein the first subset of characters and the second subset of characters further include blank symbols. 11. Any method from Sections 1 to 10, further comprising determining the proportion of blank symbols in the predicted set of characters for a segment by determining the EOS corresponding to at least one segment of the set of segments. 12. The method of Section 11, further comprising determining that the percentage has exceeded the EOS threshold by determining that the EOS corresponds to at least one segment of a set of segments. 13. The method of Section 11 or 12, further comprising determining the proportion of blank symbols in a sliding window of a first length, wherein the segments are contained within the sliding window. 14. Any method from Sections 1 to 13, wherein predicting one or more sets of characters involves using one or more neural networks to predict one or more sets of characters. 15. The method of Section 14, further comprising applying the outputs of one or more neural networks to a Connectionist Temporal Classification (CTC) function that predicts one or more sets of characters, generates a probability distribution of one of the characters in the set as an output, and provides the probability distribution to an EOS detector. 16. It is a system, One or more processors, A memory that stores instructions, the instructions being executed by one or more processors and then used by the system To generate a determination of a speech end (EOS) threshold determined at least partially based on speech rate, and To generate an EOS decision for an audio window based at least partially on the EOS threshold. To make it do, memory and A system that includes these features. 17. The system of Section 16, wherein memory further includes instructions, which, as a result of being executed by one or more processors, cause the system to obtain a set of predicted characters for an audio window generated using at least one of the following: one or more neural networks implementing a neuroacoustic model, or a Connectionist Temporal Classification (CTC) function. 18. Memory further contains instructions, and instructions are executed by one or more processors and then stored in the system. This involves obtaining a set of words from an audio signal, where the audio window represents a portion of the audio signal. Determining speech speed based at least partially on a set of word intervals determined at least partially on a set of words and A system according to section 16 or 17 that performs the following. 19. A system of any of Sections 16-18, wherein the memory further includes instructions, which, as a result of being executed by one or more processors, cause the system to determine a speech rate based at least partly on the maximum value of a set of interword intervals calculated at least partly on a set of words contained in a set of windows representing an audio signal, and the audio window is a member of the set of windows. 20. Any system from Sections 16-19, in which the audio window includes multiple time steps of an audio signal, including speech. 21. The memory further contains instructions, which, as a result of being executed by one or more processors, cause the system to execute the speech processing pipeline. Instructions causing the system to generate an EOS decision, executed by one or more processors, further include instructions causing the system to generate an EOS decision as part of the speech processing pipeline. One of the systems described in sections 16-20. 22. The system of Section 21, in which the generation of the EOS decision is performed by the EOS detector of the speech processing pipeline. twenty three. Flagging a subset of audio frames in an audio signal as End of Speech (EOS) based at least in part on an EOS threshold, wherein the EOS threshold is determined at least in part on a set of word intervals. Methods that include... 24. The method of Section 23, further comprising using a set of features of a set of audio frames generated by an acoustic model as input to determine a set of interword spacings based at least in part on a set of characters generated by a Connectionist Temporal Classification (CTC) function. 25. The method of Section 23 or 24, further comprising determining the EOS threshold by setting the EOS threshold to the average interval of a set of interword intervals. 26. Any method of Sections 23–25, further comprising generating a transcript of an audio signal as a result of flagging a subset of audio frames as EOS. 27. Any method from Sections 23-26, further comprising providing an EOS threshold to a speech recognition pipeline. 28. A processor, which is at least partially, One or more neural networks predict the set of characters associated with a segment in a set of audio input segments, One or more neural networks determine the End of Speech (EOS) threshold for a segment, at least in part, based on the determination that the End of Speech (EOS) threshold is met for the set of characters associated with the segment. A processor comprising one or more arithmetic logic units (ALUs) for generating an EOS threshold determination determined at least in part on speech rate, using one or more neural networks. 29. The processor of Section 28, further comprising one or more neural networks, comprising a neuroacoustic model. 30. The processor of Section 28 or 29, in which one or more neural networks are used to implement the speech processing pipeline. 31. One or more ALUs, using a set of features of a set of segments of audio input generated by an acoustic model, further determine a set of interword spacings, at least in part, based on a set of characters generated by a Connectionist Temporal Classification (CTC) function, according to any of the processors in Sections 28–30.

[0105] Furthermore, while subject matter has been described in terms specific to structural features and / or methodological actions, it will be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms of implementation of the claims.

[0106] Servers and data centers The following diagrams illustrate, without limitation, exemplary network server and data center-based systems that may be used to implement at least one embodiment.

[0107] Figure 12 shows a distributed system 1200 according to at least one embodiment. In at least one embodiment, the distributed system 1200 includes one or more client computing devices 1202, 1204, 1206, and 1208, which are configured to run and operate client applications such as web browsers, proprietary clients, and / or variations thereof on one or more networks 1210. For example, one or more client computing devices 1202, 1204, 1206, and 1208 run applications for running audio capture devices as described above. In at least one embodiment, a server 1212 may be connected to the remote client computing devices 1202, 1204, 1206, and 1208 via the network 1210 in a communicative manner. Furthermore, in various embodiments, the server 1212 runs audio analyzers and / or components of a speech processing pipeline as described above.

[0108] In at least one embodiment, server 1212 may be adapted to run one or more services or software applications, such as services and applications capable of managing session activity for single sign-on (SSO) access across multiple data centers. In at least one embodiment, server 1212 may also provide other services or software applications that may include non-virtual and virtual environments. In at least one embodiment, these services may be provided to users of client computing devices 1202, 1204, 1206, and / or 1208 as web-based services or cloud services, or under a Software as a Service (SaaS) model. In at least one embodiment, users operating client computing devices 1202, 1204, 1206, and / or 1208 may then interact with server 1212 and utilize the services provided by these components using one or more client applications.

[0109] In at least one embodiment, software components 1218, 1220, and 1222 of system 1200 are implemented on server 1212. In at least one embodiment, one or more components of system 1200, and / or services provided by these components, may also be implemented by one or more client computing devices 1202, 1204, 1206, and / or 1208. In at least one embodiment, a user operating a client computing device may then use one or more client applications to access the services provided by these components. In at least one embodiment, these components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that various different system configurations are possible that may differ from the distributed system 1200. The embodiment shown in Figure 12 is therefore one example of a distributed system for implementing the system of the embodiment, and is not intended to be limiting.

[0110] In at least one embodiment, client computing devices 1202, 1204, 1206, and / or 1208 may include various types of computing systems. In at least one embodiment, the client computing device may include a portable handheld device (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or a wearable device (e.g., Google Glass® head-mounted display) running software such as Microsoft Windows Mobile®, and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, and / or variations thereof. In at least one embodiment, the device may support various applications such as various internet-related apps, email, and short message service (SMS) applications, and may use various other communication protocols. In at least one embodiment, the client computing device may also include, as an example, a general-purpose personal computer, including personal computers and / or laptop computers, running various versions of the Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. In at least one embodiment, the client computing device may be a workstation computer running any of the various commercially available UNIX® or UNIX®-like operating systems, including, but not limited to, various GNU / Linux® operating systems such as Google Chrome OS.In at least one embodiment, the client computing device may also include electronic devices such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device that can communicate over network 1210. The distributed system 1200 in Figure 12 is shown using four client computing devices, but any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with server 1212.

[0111] In at least one embodiment, the network 1210 in the distributed system 1200 may be any type of network capable of supporting data communication using any of the various available protocols, including but not limited to TCP / IP (transmission control protocol / Internet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk, and / or variations thereof. In at least one embodiment, the network 1210 may be a local area network (LAN), an Ethernet-based network, Token Ring, a wide area network, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suites, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0112] In at least one embodiment, server 1212 may consist of one or more general-purpose computers, specialized server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. In at least one embodiment, server 1212 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization. In at least one embodiment, one or more flexible pools of logical storage devices may be virtualized to maintain virtual storage devices for the server. In at least one embodiment, a virtual network may be controlled by server 1212 using software-defined networking. In at least one embodiment, server 1212 may be adapted to run one or more services or software applications.

[0113] In at least one embodiment, server 1212 may run any operating system and any commercially available server operating system. In at least one embodiment, server 1212 may also run any of a variety of additional server applications and / or intermediate tier applications, including HTTP (hypertext transport protocol) servers, FTP (file transfer protocol) servers, CGI (common gateway interface) servers, JAVA® servers, database servers, and / or variations thereof. In at least one embodiment, exemplary database servers include, without limitation, commercially available database servers from Oracle, Microsoft, Sybase, IBM (International Business Machines), and / or variations thereof.

[0114] In at least one embodiment, the distributed system 1200 may also include one or more databases 1214 and 1216. In at least one embodiment, the databases may provide a mechanism for storing information such as audio, speech rate information, user information, usage pattern information, conformance rule information, and other information. In at least one embodiment, the databases 1214 and 1216 may reside in various locations. In at least one embodiment, one or more of the databases 1214 and 1216 may reside on a non-temporary storage medium local to (and / or residing on) the server 1212. In at least one embodiment, the databases 1214 and 1216 may be remote from the server 1212 and be in communication with the server 1212 via a network-based connection or a dedicated connection. In at least one embodiment, the databases 1214 and 1216 may reside on a storage area network (SAN). In at least one embodiment, any files necessary to perform functions attributable to server 1212 may be stored locally and / or remotely on server 1212, as needed. In at least one embodiment, databases 1214 and 1216 may include relational databases, such as databases adapted to store, update, and retrieve data in response to commands in SQL format.

[0115] Figure 13 shows a cloud computing environment 1302 according to at least one embodiment. In at least one embodiment, the cloud computing environment 1302 comprises one or more computer systems / servers 1304 with which computing devices such as a personal digital assistant (PDA) or cellular phone 1306A, a desktop computer 1306B, a laptop computer 1306C, and / or an automotive computer system 1306N communicate. In at least one embodiment, this allows the infrastructure, platform, and / or software to be provided as a service from the cloud computing environment 1302 so that each client does not need to maintain such resources separately. It is understood that the types of computing devices 1306A-1306N shown in Figure 13 are for illustrative purposes only, and that the cloud computing environment 1302 can communicate with any type of computerized device over any type of network and / or network / addressable connection (e.g., using a web browser). In various embodiments, computing devices 1306A to 1306N may simply capture audio and communicate the captured audio (e.g., input audio) to one or more computer systems / servers 1304. For example, an automotive computer system 1306N may transmit the captured audio directly from within the cabin to one or more computer systems / servers 1304 without processing the captured audio.

[0116] In at least one embodiment, the computer system / server 1304 can be represented as a cloud computing node and can operate in a great many other general-purpose or specialized computing system environments or configurations. In at least one embodiment, examples of computing systems, environments, and / or configurations suitable for use with the computer system / server 1304 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices and / or variations thereof.

[0117] In at least one embodiment, the computer system / server 1304 may be described in the general context of computer system executable instructions, such as program modules, which are executed by the computer system. In at least one embodiment, a program module includes routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. In at least one embodiment, the exemplary computer system / server 1304 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. In at least one embodiment, in a distributed cloud computing environment, program modules may reside in both local and remote computer system storage media, including memory storage devices.

[0118] Figure 14 shows a set of functional abstraction layers provided by the cloud computing environment 1302 (Figure 13) in at least one embodiment. It should be understood that the components, layers, and functionalities shown in Figure 14 are for illustrative purposes only, and that these components, layers, and functionalities may change.

[0119] In at least one embodiment, the hardware and software layer 1402 includes hardware and software components. In at least one embodiment, examples of hardware components include mainframes, various RISC (Reduced Instruction Set Computer) architecture-based servers, various computing systems, supercomputing systems, storage devices, networks, networking components, and / or variations thereof. In at least one embodiment, examples of software components include network application server software, various application server software, various database software, and / or variations thereof.

[0120] In at least one embodiment, the virtualization layer 1404 provides an abstraction layer from which exemplary virtual entities such as virtual servers, virtual storage, virtual networks (including virtual private networks), virtual applications, virtual clients, and / or variations thereof may be provided.

[0121] In at least one embodiment, the management layer 1406 provides various functions. In at least one embodiment, resource provision provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. In at least one embodiment, metering provides usage tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In at least one embodiment, resources may include application software licenses. In at least one embodiment, security provides identity verification for users and tasks, as well as protection of data and other resources. In at least one embodiment, the user interface provides access to the cloud computing environment for both users and system administrators. In at least one embodiment, service level management allocates and manages cloud computing resources to ensure that required service levels are met. In at least one embodiment, service level agreement (SLA) management provides pre-placement and procurement of cloud computing resources for which future requirements are anticipated according to the SLA.

[0122] In at least one embodiment, workload layer 1408 provides functionality that utilizes a cloud computing environment. In at least one embodiment, examples of workloads and functions that may be provided from this layer include mapping and navigation, software development and management, educational services, data analysis and processing, transaction processing, and service delivery. For example, workload layer 1408 may perform all or part of a speech processing pipeline as described above.

Claims

1. A step of determining an End of Speech (EOS) threshold based at least partially on speech speed, A step of predicting a set of characters representing at least one segment from a set of segments of an audio input, A step of determining the EOS corresponding to the at least one segment, at least in part, based on the determination that the EOS threshold is met for the set of predicted characters for the at least one segment. Includes, A method in which the speech rate is determined at least in part on a set of word intervals, the set of word intervals being calculated by determining at least the time between a first subset of characters in the set of characters and a second subset of characters in the set of characters.

2. The method according to claim 1, wherein the set of predicted characters for the segment is generated by a connectionist temporal classification (CTC) function generating as an output the probability distribution of one character from the set of characters.

3. The method according to claim 1, wherein the step of determining the EOS threshold further includes the step of modifying the EOS threshold based at least in part on the maximum interword interval of the set of interword intervals.

4. The method according to claim 1, wherein the step of determining the EOS threshold further includes the step of modifying the EOS threshold based at least in part on the median value of the set of interword intervals.

5. The method according to claim 1, wherein the step of determining the EOS threshold further includes the step of modifying the EOS threshold based at least in part on the distribution of the set of interword intervals and the maximum interword interval of the set of interword intervals.

6. The method according to claim 2, wherein the set of characters is determined by applying at least a greedy selection algorithm to the output of the CTC function.

7. The method according to claim 6, wherein the first subset of characters and the second subset of characters further include blank symbols.

8. The method according to claim 1, wherein the step of determining the EOS corresponding to at least one segment of the set of segments further includes the step of determining the proportion of blank symbols in the set of predicted characters for the segment.

9. The method according to claim 8, wherein the step of determining the EOS corresponding to at least one segment of the set of segments further includes the step of determining that the proportion exceeds the EOS threshold.

10. The method according to claim 8, wherein the step of determining the proportion of blank symbols further includes the step of determining the proportion of blank symbols in a sliding window of a first length, and the segment is included in the sliding window.

11. The method according to claim 1, wherein the step of predicting the set of one or more characters includes the step of predicting the set of one or more characters using one or more neural networks.

12. The method according to claim 11, wherein the step of predicting the set of one or more characters further includes the step of applying the output of the one or more neural networks to a Connectionist Temporal Classification (CTC) function that generates a probability distribution of one of the characters from the set of characters as an output and provides the probability distribution to an EOS detector.

13. It is a system, One or more processors, A memory for storing instructions, wherein the instructions, as a result of being executed by one or more processors, are used in the system. To generate a determination of a speech termination (EOS) threshold determined at least partially based on speech rate, and To generate an EOS determination for an audio window based at least partially on the aforementioned EOS threshold. To make it do, memory and Equipped with, A system in which the speech rate is determined at least in part on a set of word intervals, the set of word intervals being calculated by determining at least the time between a first subset of characters in the set of characters and a second subset of characters in the set of characters.

14. The system according to claim 13, wherein the memory further includes instructions, which, as a result of being executed by the one or more processors, cause the system to obtain a set of predicted characters for the window of audio generated using at least one of a neural network implementing a neuroacoustic model or a Connectionist Temporal Classification (CTC) function.

15. The memory further includes instructions, which, as a result of being executed by the one or more processors, are transmitted to the system. The process of obtaining a set of words from an audio signal, wherein the audio window represents a portion of the audio signal. Determining the speech rate based at least partially on a set of interword spacings determined at least partially on the set of words The system according to claim 13, which causes the following to be performed.

16. The system according to claim 13, wherein the memory further includes instructions, which, as a result of being executed by the one or more processors, cause the system to determine the speech rate based at least in part on the maximum value of a set of interword intervals calculated at least in part on a set of words included in a set of windows representing an audio signal, and the window of audio is a member of the set of windows.

17. The system according to claim 13, wherein the audio window includes a plurality of time steps of an audio signal, including speech.

18. The memory further includes instructions, which, as a result of being executed by the one or more processors, cause the system to execute a speech processing pipeline. The instruction causing the system to generate the decision of the EOS further includes an instruction causing the system to generate the decision of the EOS as part of the speech processing pipeline, as a result of the execution of the one or more processors. The system according to claim 13.

19. The system according to claim 18, wherein generating the determination of the EOS is performed by the EOS detector of the speech processing pipeline.

Images