Detecting voice interruptions for conversational AI systems and applications

Abstract

The invention relates to detecting voice interruptions for conversational AI systems and applications. In various examples, voice interrupt detection for conversational AI systems and applications is described herein. Systems and methods disclosed herein use both sentence end detection and utterance end detection associated with words (e.g., tags) in text to determine when to further process various portions of the text. For example, one or more models may process text data associated with text, where the text data may be generated using an Automatic Speech Recognition (ARS) model based on audio data representing speech. Based at least on the processing of the textual data, the model may generate and / or output data representing a first indicator that the word is associated with the end of the sentence, a second indicator that the word is associated with the end of the utterance, and a third indicator that the word is not associated with the end of the sentence or the end of the utterance.

Description

BACKGROUND

[0001] Automatic speech recognition (ASR) can play an important role in conversational artificial intelligence (Al) systems and applications. For example, ASR can convert audio data representing speech into tokens, such as tokens representing individual portions of words (e.g., sub-words), complete words, punctuation, symbols, letters, and the like corresponding to the speech. Embeddings associated with these tokens can then be applied to one or more language models, such as one or more natural language models or one or more neural machine translation models, for further processing. In some cases, before applying the embeddings, it can be determined when a user has finished speaking so that the entire text associated with the utterance can be applied to the language model(s). For example, utterance end detection can be used to attempt and accurately detect when a user has finished speaking. However, in some cases, an utterance can contain multiple sentences, in which case the accuracy of the language model can improve when the portions of the utterance associated with the sentences are processed separately. While these traditional systems are able to detect the end of an utterance, these traditional systems performing utterance end detection can not be able to detect the end of the sentences.

[0002] Accordingly, other techniques have been used to detect the end of sentences associated with an utterance. For example, some traditional systems can detect the end of a sentence based on detecting capitalization of words in the utterance. However, in some cases, words contained in the middle of a sentence can be capitalized, such as when the words contain names and / or specific locations. Further, other traditional systems can detect the end of a sentence based on punctuation contained in the utterance, such as periods and question marks. However, in some cases, punctuation, such as periods, can be contained in the middle of a sentence, such as in “Mr.” or “Mrs.” Further, other traditional systems can detect the end of a sentence based on pauses in the utterance. However, in some cases, based on different ways in which users speak, it can be difficult to detect pauses between sentences. Further, sentence end detection techniques are unable to detect the end of an utterance, which is still important for processing text associated with the utterance. SUMMARY

[0003] Embodiments of the present disclosure relate to detecting speech breaks for conversational AI systems and applications. The systems and methods disclosed herein use both end-of-sentence detection and end-of-utterance detection associated with words in text to determine when portions of the text are to be further processed. For example, one or more models can process text data associated with the text (e.g., text data representing tokens corresponding to the text), where the text data can be generated using an automatic speech recognition (ARS) model and generated based at least on audio data representing speech. Based at least on the processing of the text data, the models can generate and / or output data representing: a first indicator that a word (and / or token) is associated with an end-of-sentence; a second indicator that a word (and / or token) is associated with an end-of-utterance; and a third indicator that a word (and / or token) is not associated with either an end-of-sentence or an end-of-utterance. This output can then be used to detect locations of end-of-sentence and / or end-of-utterance, where these locations result in portions of the text being further processed. In some examples, the systems and methods can use one or more additional techniques to determine these locations, such as one or more models that perform case detection and / or one or more models that perform punctuation detection.

[0004] In contrast to conventional systems, the systems of the present disclosure can use both end-of-sentence detection and end-of-utterance detection to determine locations in text (e.g., tokens) for further processing at the same time. In this way, the systems of the present disclosure are able to detect end-of-sentence when an utterance contains multiple sentences and detect the end of the utterance, which can improve the performance of a language model by processing particular portions of the text in comparison to conventional systems that use only case detection, punctuation detection, and / or pause detection. Moreover, as will be described in greater detail herein, the systems of the present disclosure use models that are trained to detect both the end of a sentence and the end of an utterance, which can improve the performance of the models. BRIEF DESCRIPTION OF DRAWINGS

[0005] The present systems and methods for detecting speech breaks for conversational AI systems and applications are described in detail below with reference to the attached drawing figures, wherein:

[0006] Figure 1 An example of a first process of performing speech break detection is shown in accordance with some embodiments of the present disclosure;

[0007] Figure 2 An example of performing end-of-sentence detection and end-of-utterance detection associated with speech is shown in accordance with some embodiments of the present disclosure;

[0008] Figure 3 Another example of performing end-of-sentence detection and end-of-utterance detection associated with speech is shown in accordance with some embodiments of the present disclosure;

[0009] Figure 4 A dataflow diagram illustrating a process for training one or more models to perform speech interruption detection, in accordance with some embodiments of the disclosure is shown;

[0010] Figure 5 An example of a second process for performing speech interruption detection using one or more additional models, in accordance with some embodiments of the disclosure is shown;

[0011] Figures 6A-6B An example of performing case detection associated with speech, in accordance with some embodiments of the disclosure is shown;

[0012] Figures 7A-7B An example of performing punctuation detection associated with speech, in accordance with some embodiments of the disclosure is shown;

[0013] Figure 8 A dataflow diagram illustrating another process for training one or more models to perform speech interruption detection, in accordance with some embodiments of the disclosure is shown;

[0014] Figure 9 An example of a third process for performing speech interruption detection using fusion, in accordance with some embodiments of the disclosure is shown;

[0015] Figure 10 An example of a system that can perform one or more processes described herein, in accordance with some embodiments of the disclosure is shown;

[0016] Figure 11 A flow diagram illustrating a method for determining at least sentence endings and utterance endings associated with tokens, in accordance with some embodiments of the disclosure is shown;

[0017] Figure 12 A flow diagram illustrating a method for performing speech interruption detection, in accordance with some embodiments of the disclosure is shown;

[0018] Figure 13 A flow diagram illustrating a method for determining sentence endings and utterance endings associated with speech, in accordance with some embodiments of the disclosure is shown;

[0019] Figure 14A is a block diagram of an example generative language model system suitable for implementing some embodiments of the disclosure;

[0020] Figure 14B is a block diagram of an example generative language model including a transformer encoder-decoder suitable for implementing some embodiments of the disclosure;

[0021] Figure 14C is a block diagram of an example generative language model including a decoder-only transformer architecture suitable for implementing some embodiments of the disclosure;

[0022] Figure 15 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0023] Figure 16 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Systems and methods related to detecting speech breaks for dialog systems and applications are disclosed. For example, a system can generate, obtain, receive, determine, and / or retrieve audio data representing speech from a user. As described herein, the speech can be associated with an utterance, such as an utterance containing one or more sentences. For example, the speech can be associated with an utterance such as "Good morning, John. How are you?" The system can then process the audio data using one or more techniques to generate text data associated with the speech. For example, in some examples, the system can process the audio data using one or more models associated with automatic speech recognition (ASR), referred to as "first models" in some examples, to generate the text data. Thus, in some examples, the text data can be associated with a text transcription of the speech, such as by representing words, styles, punctuation, etc. that match the speech. Additionally, in some examples, the text can be represented using one or more tokens, such as one or more tokens representing portions of words, words, punctuation, symbols, letters, etc. in the text.

[0025] In some examples, the system can further process the text data, such as by using one or more language models. As described herein, a language model can include, but is not limited to, one or more natural language models, one or more neural machine translation models, one or more large language models, and / or any other type of language model. In some examples, for example to improve the performance of the language model processing the text data, the system can be configured to determine one or more locations in the text associated with a break in order to process the text data, such as the end of one or more sentences in the text and / or the end of one or more utterances. For example, if the utterance includes two sentences, such as similar to the example above, the system can process a first portion of the text data associated with the first sentence (e.g., "Good morning, John") in a first instance and then process at least a second portion of the text data associated with the second sentence (e.g., "How are you") and / or the entire utterance (e.g., "Good morning, John. How are you?") in a second instance.

[0026] For example, the system can process textual data using one or more encoders (e.g., one or more text encoders) configured to generate one or more embeddings corresponding to the text (e.g., tokens). The system can then apply data representing the embeddings to one or more models (also referred to as “second models” in certain examples) trained to perform end-of-sentence (EOS) detection and end-of-utterance (EOU) detection. For example, based at least on processing of the data, the second models can generate and / or output data representing one or more first indicators of words in the text (and / or tokens representing the text) associated with an EOS, one or more second indicators of words in the text (and / or tokens representing the text) associated with an EOU, and / or one or more third indicators of words in the text (and / or tokens representing the text) not associated with an EOS or an EOU (e.g., also referred to as “normal words”). In some examples, the indicators can be associated with probabilities of the words being associated with an EOS, an EOU, or a normal word. For example, for a word, the output data can represent at least a first probability of the word being associated with an EOS, a second probability of the word being associated with an EOU, and a third probability of the word being associated with a normal word.

[0027] The system (and / or the second models) can then use the indicators to determine one or more locations in the text associated with an EOS or an EOU. For a first example, and using the above example of a word being associated with three probabilities, the system can determine that the word is associated with an EOS location based at least on the first probability containing the highest probability and / or the first probability satisfying a threshold probability. For a second example, and again using the above example of a word being associated with three probabilities, the system can determine that the word is associated with an EOU location based at least on the second probability containing the highest probability and / or the second probability satisfying a threshold probability. For a third example, and again using the above example of a word being associated with three probabilities, the system can determine that the word is associated with a normal word location (e.g., not associated with an EOS location or an EOU location) based at least on the third probability containing the highest probability, the third probability satisfying a threshold probability, and / or the first probability and / or the second probability not satisfying a threshold probability. While these are just some example techniques by which the system can detect EOS and / or EOU locations, in other examples, the system can use other and / or alternative techniques.

[0028] In some examples, the system can perform one or more additional techniques to detect EOS and / or EOU locations in the text. For instance, in some examples, the system can apply the data representing the embeddings to one or more models trained to perform case detection (also referred to as“third models” in some examples). For example, based at least on processing of the data, the third models can generate and / or output data representing one or more first indicators that a word in the text (and / or a token representing the text) is associated with a lower case word and / or one or more second indicators that a word in the text (and / or a token representing the text) is associated with an upper case word. In some examples, these indicators can be associated with probabilities that the word is associated with a lower case word and / or an upper case word. For instance, for a word, the output data can represent at least a first probability that the word is associated with a lower case word and a second probability that the word is associated with an upper case word.

[0029] The system can then use these indicators to further identify EOS and / or EOU locations in the text. For the first example, and using the above example where a word is associated with two probabilities, when the second probability contains the highest probability and / or satisfies a threshold probability, the system can determine that the previous word is more likely to be associated with an EOS location. In some examples, this can be because the third model indicates that the word contains an upper case word, which can indicate the beginning of a new sentence, such that the previous word is the end of the previous sentence. For the second example, and again using the above example where a word is associated with two probabilities, when the first probability contains the highest probability and / or satisfies a threshold probability, the system can determine that the previous word is less likely to be associated with an EOS location and / or an EOU location. In some examples, this can be because the third model indicates that the word contains a lower case word, which can indicate the middle of a sentence. While these are just a few example techniques by which the system can use case detection to further detect EOS and / or EOU locations, in other examples, the system can use additional and / or alternative techniques.

[0030] Additionally, or alternatively, in some examples, the system can apply the data representing the embeddings to one or more models trained to perform punctuation detection (also referred to as “fourth models” in some examples). For example, based at least on processing of the data, the fourth models can generate and / or output data representing one or more first indicators that a word in the text (and / or a token representing the text) is associated with one or more types of punctuation and / or one or more second indicators that a word in the text (and / or a token representing the text) is not associated with one or more punctuation. In some examples, these indicators can be associated with a word and a probability of being associated with a certain type of punctuation and / or not being associated with punctuation. For example, for a word, the output data can represent at least a first probability that the word is associated with a first type of punctuation (e.g., a period), a second probability that the word is associated with a second type of punctuation (e.g., a comma), and so on, up to a last probability that the word is not associated with punctuation.

[0031] The system can then use these indicators to further identify EOS and / or EOU locations in the text. For a first example, when a probability associated with an end-of-sentence punctuation (e.g., a period, a question mark, an exclamation point, etc.) includes a highest probability and / or satisfies a threshold probability, the system can determine that the word is more likely to be associated with an EOS location and / or an EOU location. For a second example, when a probability associated with an in-sentence punctuation (e.g., a comma, etc.) includes a highest probability and / or satisfies a threshold probability, the system can determine that the word is less likely to be associated with an EOS location and / or an EOU location. Still, for a third example, when a probability associated with not being associated with punctuation includes a highest probability and / or satisfies a threshold probability, the system can determine that the word is less likely to be associated with an EOS location and / or an EOU location. While these are just some example techniques by which the system can use punctuation detection to further detect EOS and / or EOU locations, in other examples, the system can use other and / or alternative techniques.

[0032] Additionally, or alternatively, in some examples, the system can further process the audio data using one or more encoders (e.g., one or more audio encoders) to generate one or more additional embeddings associated with the speech. The system can then perform one or more fusion techniques to fuse the embeddings associated with the text and the additional embeddings associated with the speech to generate one or more input embeddings. For example, the system can then input data representing the input embeddings into the second model, the third model, and / or the fourth model in addition to or instead of the embeddings associated with the text.

[0033] As described herein, in some examples, the system can perform one or more techniques to train the second model, the third model, and / or the fourth model. For a first example, if the system uses only the second model, the system can generate, obtain, receive, determine, and / or retrieve training data representing text instances associated with various utterances, and truth data representing indicators associated with the text instances, such as indicators indicating whether a word is associated with an EOS, an EOU, and / or a normal word. Then, the system can apply data associated with the text instances (e.g., data representing embeddings associated with tokens corresponding to the text) into the second model processing the data, and generate, based at least on the processing, outputs indicating the indicators for the text instances. Further, the system can determine one or more losses using the truth indicators and the output indicators. Then, the system can update the second model using the losses.

[0034] For a second example, if the system uses the second model, the third model, and / or the fourth model, the system can generate, obtain, receive, determine, and / or retrieve training data representing text instances associated with various utterances, and truth data representing indicators associated with the text instances. For example, the indicators can indicate whether a word is associated with an EOS, an EOU, a normal word, a lower case word, an upper case word, contains a punctuation, and / or does not contain a punctuation. Then, the system can apply data associated with the text instances (e.g., data representing embeddings associated with tokens corresponding to the text) into the second model, the third model, and / or the fourth model, which process the data, and generate, based at least on the processing, outputs indicating the indicators for the text instances, as described herein. Further, the system can determine one or more losses using the truth indicators and the output indicators. Then, the system can update the second model, the third model, and / or the fourth model using the losses.

[0035] While examples herein describe detecting EOS and / or EOU associated with words in text, in other examples, similar processes can be used to detect EOS and / or EOU associated with tokens corresponding to the text. For example, similar processes can be used to determine indicators for tokens using a second model, a third model, and / or a fourth model. These indicators can then be used to determine EOS and EOU using one or more processes described herein. Further, the text data representing the tokens can then be divided into portions using the locations of the EOS and / or EOU for further processing. While one, two, three, four, or more models are described herein, this does not mean that discrete models must be used, but rather that there can be discrete models, or different layers (or heads, e.g., layer sets) can exist for each different task. For example, there can be a punctuation head, an EOS head, an EOU head, a capitalization head, etc. without departing from the scope of the present disclosure.

[0036] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driving assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, aircraft, watercraft, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, underwater vehicles, drones, and / or other types of vehicles. Further, the systems and methods described herein can be used for various uses, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0037] The disclosed embodiments can be included in various different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart regional monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing visual language models (VLMs), systems for implementing multi-modal language models, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing optical transport simulation, systems for performing collaborative content creation of 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0038] Referring to Figure 1 , Figure 1 An example of a first process 100 of performing speech disruption detection is shown in accordance with some embodiments of the present disclosure. It should be understood that the arrangements and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, order, function grouping, etc.) can be used in addition to or instead of the arrangements and elements illustrated, and certain elements can be wholly omitted. Moreover, many of the elements described herein are functional entities, which can be implemented as discrete or distributed components, or in conjunction with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by an entity can be performed by hardware, firmware, and / or software. For instance, various functions can be performed by a processor executing instructions stored in memory.

[0039] Process 100 can include applying audio data 102 to one or more recognition models 104. As described herein, audio data 102 can represent speech, such as an utterance containing one or more sentences from one or more users. For example, audio data 102 can represent an utterance such as “Good morning, John. How are you?” Moreover, recognition models 104 can include any type of machine learning model, neural network, component, module, software, hardware, and / or the like configured to process audio data 102 and generate, based at least on the processing, text data 106 associated with text corresponding to the speech. For example, recognition models 104 can include one or more automatic speech recognition models configured to generate text data 106 associated with a speech transcription. As such, text data 106 can represent one or more text tokens associated with text, such as text tokens representing different letters, numbers, punctuation, word parts, words, symbols, and / or the like associated with the speech.

[0040] In some examples, process 100 can subsequently include applying text data 106 to one or more language models 108. As described herein, language models 108 can include, but are not limited to, one or more natural language models, one or more neural machine translation models, one or more large / vision / multimodal language models, and / or any other type of language model configured to process text data 106 to generate output data 110 associated with the speech. For example, an utterance represented by audio data 102 can include a query, a request, a command, a question, an instruction, and / or any other type of utterance, while output data 110 can represent a response, information, another question, and / or the like associated with the utterance. For example, using the above example in which the utterance contains “Good morning, John. How are you?” output data 110 can represent a response to the utterance, such as “I am doing well.”

[0041] As described herein, in some examples, portions of the text data 106 can be applied to the language model 108 at different time instances using position interrupts associated with the text (e.g., tokens) in order to, for example, improve the performance of the language model 108 processing the text data 106. For example, portions of the text data 106 associated with one or more sentence endings and / or one or more utterance endings in the text can be applied to the language model 108 at different time instances. For a first example, using the example above, a first portion of the text data 106 representing a first sentence (e.g., “Good morning, John”) can be applied to the language model 108 at a first time instance, and then a second portion of the text data 106 representing a second sentence (e.g., “How are you”) can be applied to the language model 108 at a second time instance. For a second example, and again using the example above, a first portion of the text data 106 representing a first sentence (e.g., “Good morning, John”) can be applied to the language model 108 at a first time instance, and then a second portion of the text data 106 representing an entire utterance (e.g., “Good morning, John. How are you?”) can be applied to the language model 108 at a second time instance.

[0042] Accordingly, the process 100 can include processing at least a portion of the text data 106 using one or more encoders 112 (such as one or more text encoders) configured to generate one or more embeddings 114 (e.g., one or more vectors) representing the text data 106. For example, if the text data 106 represents tokens, the embeddings 114 can be generated by the encoders 112 to represent the tokens. While the example above describes the encoders 112 generating the embeddings 114, in other examples, the process 100 can not include using encoders 112. Rather, the process 100 can include further processing the text data 106 using one or more techniques described herein (e.g., the model can include one or more encoders that process the text data 106). Figure 1 Accordingly, the process 100 can include processing at least a portion of the text data 106 using one or more encoders 112 (such as one or more text encoders) configured to generate one or more embeddings 114 (e.g., one or more vectors) representing the text data 106. For example, if the text data 106 represents tokens, the embeddings 114 can be generated by the encoders 112 to represent the tokens. While the example above describes the encoders 112 generating the embeddings 114, in other examples, the process 100 can not include using encoders 112. Rather, the process 100 can include further processing the text data 106 using one or more techniques described herein (e.g., the model can include one or more encoders that process the text data 106).

[0043] For example, as shown, the process 100 can include applying the embedding 114 to one or more detection models 116. As described herein, in some examples, the detection models 116 can include, without limitation, one or more machine learning models, one or more neural networks, one or more layers of one or more models (e.g., one or more connection layers, one or more output layers, etc.), one or more heads of one or more models, software, hardware, and / or any other type of processing component configured to perform at least a portion of the processes described herein. For example, based at least on processing of the embedding 114, the process 100 can include the detection models 116 generating and / or outputting output data 118 associated with the speech. For example, as shown, the output data 118 can represent at least one or more word indicators 120 (e.g., normal words not associated with an end-of-sentence and / or an end-of-utterance) associated with the speech, one or more end-of-sentence (EOS) indicators 122 associated with the speech, and / or one or more end-of-utterance (EOU) indicators 124 associated with the speech.

[0044] As described herein, in some examples, the output data 118 can represent respective word indicators 120, EOS indicators 122, and EOU indicators associated with one or more (e.g., each) word in the speech (and / or one or more (e.g., each) token represented by the text data 106). For example, using the example described above, the output data 118 can represent a first word indicator 120, a first EOS indicator 122, and a first EOU indicator 124 associated with the first word “Good,” a second word indicator 120, a second EOS indicator 122, and a second EOU indicator 124 associated with the second word “morning,” a third word indicator 120, a third EOS indicator 122, and a third EOU indicator 124 associated with the third word “John,” and so on, for the remaining words included in the utterance.

[0045] As described herein, in some examples, the word indicators 120, the EOS indicators 122, and / or the EOU indicators 124 can be associated with various probabilities. For example, and again using the above example, the first word indicator 120 can include a first probability that the first word is associated with a normal word, the first EOS indicator 122 can include a second probability that the first word is associated with an EOS, and the first EOU indicator 124 can include a third probability that the first word is associated with an EOU. Further, the second word indicator 120 can include a fourth probability that the second word is associated with a normal word, the second EOS indicator 122 can include a fifth probability that the second word is associated with an EOS, and the second EOU indicator 124 can include a sixth probability that the second word is associated with an EOU. In such examples, the sum of these probabilities can amount to a maximum probability, such as 100% (and / or any other probability). For example, the first probability can include 95%, the second probability can include 2%, and the third probability can include 3%.

[0046] For example, Figure 2 An example of performing sentence end and utterance end detection associated with speech is shown in accordance with some embodiments of the disclosure. As shown, data 202 (embeddings) corresponding to the utterance "Good morning, John. How are you?" can be applied to the detection model 116. Thus, the detection model 116 can perform one or more processes described herein to generate output data 204 (which can be similar to, and / or representative of, the output data 118) associated with the data 202. For example, as shown, the output data 204 can represent probabilities 206(1)-206(6) that the six words are associated with a normal word, probabilities 208(1)-208(6) that the six words are associated with an EOS, and probabilities 210(1)-210(6) that the six words are associated with an EOU. For example, the probability 206(3) can include 2% that the third word "John." is associated with a normal word, the probability 208(3) can include 95% that the third word is associated with an EOS, and the probability 210(3) can include 3% that the third word is associated with an EOU.

[0047] While Figure 2 The example of "Good morning, John. How are you?" illustrates that the text includes capitalization and punctuation, but in other examples, the text associated with the data 202 can not include capitalization and / or punctuation. For example, the text can include "good morning john how are you". Further, while the example of "Good morning, John. How are you?" illustrates that the text includes a period, in other examples, the text associated with the data 202 can not include a period. For example, the text can include "good morning john how are you". Figure 2The example illustrates determining output data 204 (e.g., probabilities) associated with individual words in the text, but in other examples, a similar process can be used to determine probabilities associated with individual tokens corresponding to the text. For example, probability 206(1) could be associated with a token corresponding to a normal word, probability 208(1) could be associated with a token corresponding to an EOS word, and probability 210(1) could be associated with a token corresponding to an EOU word.

[0048] Return to reference Figure 1 For example, process 100 may include one or more detection components 126 processing output data 118 to detect the EOS and / or EOU locations associated with speech. As described herein, in some examples, detection component 126 may include, but is not limited to: one or more machine learning models, one or more neural networks, one or more layers of one or more models (e.g., one or more connection layers, one or more output layers, etc.), one or more heads of one or more models, one or more modules, software, hardware, and / or any other type of processing component configured to perform at least a portion of the process described herein. Furthermore, in some examples, detection component 126 may detect the EOS location at least based on probabilities associated with the EOS indicator 122, including the highest probability and / or satisfying (e.g., equal to or greater than) a threshold probability, where the threshold probability may be represented by threshold data 128. Furthermore, in some examples, detection component 126 may detect the EOU location at least based on probabilities associated with the EOU indicator 124, including the highest probability and / or satisfying (e.g., equal to or greater than) a threshold probability. As described herein, the threshold probability may include, but is not limited to, 75%, 95%, 99% and / or any other percentage.

[0049] For example, Figure 3Another example of performing sentence end and utterance end detection associated with speech is shown in accordance with some embodiments of the present disclosure. As shown, the detection component 126 can process the output data 204 and based at least on this processing, output data 302 indicative of the detection associated with the words in the speech. For example, the detection component 126 can determine that the probability 206(1) contains the highest probability of the probabilities 206(1), 208(1), and 210(1), and / or that the probability 206(1) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the first word contains a normal word 304. Then, the detection component 126 can determine that the probability 206(2) contains the highest probability of the probabilities 206(2), 208(2), and 210(2), and / or that the probability 206(2) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the second word contains a normal word 306. Then, the detection component 126 can determine that the probability 208(3) contains the highest probability of the probabilities 206(3), 208(3), and 210(3), and / or that the probability 208(3) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the third word contains an EOS position 308.

[0050] Then, the detection component 126 can determine that the probability 206(4) contains the highest probability of the probabilities 206(4), 208(4), and 210(4), and / or that the probability 206(4) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the fourth word contains a normal word 310. Then, the detection component 126 can determine that the probability 206(5) contains the highest probability of the probabilities 206(5), 208(5), and 210(5), and / or that the probability 206(5) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the fifth word contains a normal word 312. Then, the detection component 126 can determine that the probability 210(6) contains the highest probability of the probabilities 206(6), 208(6), and 210(6), and / or that the probability 210(6) satisfies a threshold probability. Accordingly, the detection component 126 can determine that the sixth word contains an EOU position 314.

[0051] In some examples, the output of the detection component 126 can include one or more letters, numbers, characters, punctuation, and / or any other type of identifier. For example, the normal word 304, 306, 310, and 312 indicators can include a first identifier, the EOS position 308 indicator can include a second identifier, and the EOU position 314 indicator can include a third identifier.

[0052] Referring back to Figure 1As an example, process 100 may include causing language model 108 to process text data 106 based at least on detections from detection component 126. For example, as described herein, when an EOS position and / or an EOU position is detected, at least a portion of text data 106 may be applied to language model 108 for processing. In some examples, applying text data 106 to language model 108 may be performed using one or more techniques. For a first example, when an EOS position is detected, a portion of text data 106 representing a sentence may be applied to language model 108, while when an EOU position is detected, the entire text data 106 representing a utterance may be applied to language model 108. For a second example, each time an EOS position and / or an EOU position is detected, a portion of text data 106 representing the first word to the specific word associated with the EOS position and / or the EOS position may be applied to language model 108. While these are just a few example techniques of how text data 106 may be applied to language model 108, in other examples, other and / or alternative techniques may be used to apply text data 106.

[0053] Although Figure 1 The example illustrates that the recognition model 104, encoder 112, detection model 116, and detection component 126 are separated from each other, but in other examples, the recognition model 104, encoder 112, detection model 116, and detection component 126 can be combined together, such as into one or more machine learning models. For example, a model may include at least the layers of encoder 112, detection model 116, and / or detection component 126.

[0054] In addition, although Figure 1The example describes the execution of process 100 when the user speaks a single language, but in other examples, process 100 can be applied to multiple languages. For example, if the user switches between different languages, a first recognition model 104 can be configured to generate first text data 106 corresponding to the first language, and a second recognition model 104 can be configured to generate second text data 106 corresponding to the second language (e.g., using code that causes automatic switching between recognition models 14). In such examples, a detection model 116 can be trained to process the first text data 106 to generate first output data 118, which includes one or more first word indicators 120 associated with the first language, one or more first sentence indicators 122 associated with the first language, and / or one or more first utterance indicators 124 associated with the first language. Furthermore, the detection model can be trained to process the second text data 106 to generate second output data 118, which includes one or more second word indicators 120 associated with the second language, one or more second sentence indicators 122 associated with the second language, and / or one or more second utterance indicators 124 associated with the second language. Thus, even if the user speaks a different language, process 100 can still be used to detect EOS and / or EOU associated with the utterance.

[0055] In addition, although Figure 1 The example describes process 100 being performed while a single user is speaking, but in other examples, process 100 may operate while multiple users are speaking. For example, recognition model 104 may generate first text data 106 associated with a first speech from a first user (e.g., the primary user) and / or second text data 106 associated with a second speech from a second user (e.g., the second user, a distracting user, a background user, etc.). Detection component 116 may then process the first text data 106 to generate first output data 118, which includes one or more first word indicators 120 associated with the first speech, one or more first sentence indicators 122 associated with the first speech, and / or one or more first utterance indicators 124 associated with the first speech. Additionally or alternatively, the detection model may process the second text data 106 to generate second output data 118, which includes one or more second word indicators 120 associated with the second speech, one or more second sentence indicators 122 associated with the second speech, and / or one or more second utterance indicators 124 associated with the second speech. Thus, method 100 can still be used to detect EOS and / or EOU associated with multiple utterances from multiple users.

[0056] In other words, the process 100 (and / or similarly, the process 500 described with respect to Figure 5 examples) can be performed when any number of users are speaking, when any language is being spoken, and / or when one or more users are speaking different languages.

[0057] Figure 4 A dataflow graph illustrating a process 400 for training a detection model 116 to perform interruption detection is shown in accordance with some embodiments of the present disclosure. As shown, the detection model 116 can be trained using training text data 402. In some examples, the training text data 402 can include text instances associated with utterances, which can be similar to and / or contain the text data 106. The training text data 402 can be synthetically generated (e.g., generated from a computer model or rendering), truthfully generated (e.g., designed and generated from real-world data such as audio data), machine-automated generated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated generated (e.g., a labeler or annotation expert defines the location of labels), and / or a combination thereof.

[0058] The detection model 116 can be trained using the training text data 402 and corresponding ground truth data 404. The ground truth data 404 can include annotations, labels, masks, and the like. For example, as shown, the ground truth data 404 can include one or more word indicators 406, one or more EOS indicators 408, and / or one or more EOU indicators 410 associated with the text instances. The ground truth data 404 can be synthetically generated (e.g., generated by a computer model or rendering graph), truthfully generated (e.g., designed and generated from real-world data), machine-automated generated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated generated (e.g., a labeler or annotation expert defines the location of labels), and / or a combination thereof. In some examples, there can be corresponding ground truth data 404 for each instance of the training text data 402.

[0059] As Figure 4Examples further illustrate that the training engine 412 can use one or more loss functions that measure a loss (e.g., an error) of the output 414 compared to the ground truth data 404. In some examples, the output 414 can be similar to and / or contain the output data 118. For example, the output 414 can contain at least one or more word indicators 416 associated with the text instance, one or more EOS indicators 418, and / or one or more EOU indicators 420. Any type of loss function can be used, such as a cross-entropy loss, a mean squared error, a mean absolute error, a mean bias error, and / or other types of loss functions. In some examples, different outputs 414 can have different loss functions. For example, the word indicators 416 can include a first loss function and / or a first loss, the EOS indicators 418 can include a second loss function and / or a second loss, and / or the EOU indicators 420 can include a third loss function and / or a third loss. In these examples, the loss functions can be combined to form a total loss (wherein the one or more losses can be weighted), and the total loss can be used to train the detection model 116 (e.g., update the parameters of the detection model 116). In any examples, backpropagation calculations can be performed to recursively calculate gradients of the loss functions with respect to the training parameters. In certain examples, these gradients can be calculated using weights and / or biases of the detection model 116.

[0060] Figure 5 An example of a second process 500 that performs speech interruption detection using one or more additional models is shown in accordance with some embodiments of the disclosure. As shown, in addition to the process 100, the process 500 can include applying the embedding 114 to one or more case model(s) 502. As described herein, in some examples, the case model(s) 502 can include, without limitation, one or more machine learning models, one or more neural networks, one or more layers of one or more models (e.g., one or more connection layers, one or more output layers, etc.), one or more heads of one or more models, software, hardware, and / or any other type of processing component configured to perform at least a portion of the processes described herein. For example, based at least on processing of the embedding 114, the process 500 can include the case model(s) 502 generating and / or outputting output data 504 associated with the speech. For example, as shown, the output data 504 can represent at least one or more lower case indicators 506 associated with the speech and / or one or more upper case indicators 508 associated with the speech.

[0061] As described herein, in some examples, output data 504 may represent corresponding lowercase indicators 506 and uppercase indicators 508 associated with one or more (e.g., each) words (and / or one or more (e.g., each) markers represented by text data 106) from the speech. For example, using the example above, output data 504 may represent a first lowercase indicator 506 and a first uppercase indicator 508 associated with the first word “Good,” a second lowercase indicator 506 and a second uppercase indicator 508 associated with the second word “morning,” a third lowercase indicator 506 and a third uppercase indicator 508 associated with the third word “John.”, and so on, as well as for the remaining words included in the discourse.

[0062] Furthermore, in some examples, lowercase indicator 506 and / or uppercase indicator 508 can be associated with various probabilities. For example, using the example above again, the first lowercase indicator 506 can contain a first probability that the first word is lowercase, while the first uppercase indicator 508 can contain a second probability that the first word is uppercase. Furthermore, the second lowercase indicator 506 can contain a third probability that the second word is lowercase, while the second uppercase indicator 508 can contain a fourth probability that the second word is uppercase. In these examples, the sum of these probabilities can reach a specific probability, such as 100% (and / or any other probability). For example, the first probability could include 95%, while the second probability could include 5%.

[0063] For example, Figures 6A-6B Examples of performing voice-related case detection according to some embodiments of this disclosure are shown. Figure 6A As shown in the example, Figure 2 The data 202 in the example (e.g., the embedding) can also be applied to the case model 502. Therefore, the case model 502 can perform one or more of the processes described herein to generate output data 602 (which may resemble and / or represent output data 504) associated with the utterance corresponding to text 202. For example, as shown, output data 602 may represent probabilities 604(1)-604(6) that the six words are lowercase and probabilities 606(1)-606(6) that the six words are uppercase. For example, probability 604(4) may include 2% that the fourth word “How” is lowercase, while probability 606(4) may include 98% that the fourth word is uppercase.

[0064] Although Figure 6AThe example shows the determination of output data 602 (e.g., probabilities) associated with individual words in the text, but in other examples, a similar process can be used to determine the probabilities associated with individual tokens corresponding to the text. For example, probability 604(1) could be associated with a token corresponding to a lowercase word, while probability 606(1) could be associated with a token corresponding to an uppercase word.

[0065] Return to reference Figure 5 For example, process 500 may include detection component 126 further processing output data 504 to detect EOS and / or EOU positions associated with speech. For the first example, using the example of a word being associated with two probabilities above, when the second probability contains the highest probability and / or meets a threshold probability, detection component 126 may determine that the preceding word is more likely to be associated with the EOS and / or EOU position. In some examples, this might be because the case model 502 indicates that the word contains an uppercase word, which could indicate the beginning of a new sentence, thus the preceding word is the end of the previous sentence. For the second example, again using the example of a word being associated with two probabilities above, when the first probability contains the highest probability and / or meets a threshold probability, detection component 126 may determine that the preceding word is less likely to be associated with the EOS and / or EOU position. In some examples, this might be because the case model 502 indicates that the word contains a lowercase word, which could indicate the middle of a sentence. While these are just some example techniques on how detection component 126 can utilize case detection to further detect EOS and / or EOU locations, in other examples, detection component 126 may use other and / or alternative techniques.

[0066] For example, Figure 6B Another example of performing speech-related case detection according to some embodiments of the present disclosure is shown. As shown, detection component 126 can use output data 602 to generate output data 608 indicating whether a word is lowercase or uppercase. For example, detection component 126 can determine that a first word is uppercase 610 based at least on probability 606(1), including the highest probability and / or a threshold probability, determine that a second word is lowercase 612 based at least on probability 604(2), including the highest probability and / or a threshold probability, determine that a third word is uppercase 614 based at least on probability 606(3), including the highest probability and / or a threshold probability, determine that a fourth word is uppercase 616 based at least on probability 606(4), including the highest probability and / or a threshold probability, determine that a fifth word is lowercase 618 based at least on probability 604(5), including the highest probability and / or a threshold probability, and / or determine that a sixth word is lowercase 620 based at least on probability 604(6), including the highest probability and / or a threshold probability. Detection component 126 can then determine the sixth word is lowercase 620 based on the following... Figure 3The example uses output data 608 when generating output data 302.

[0067] Although Figure 6B The example illustrates the generation of output data 608 for each word, but in other examples, the detection component 126 may generate similar output data 608 for each token associated with the text corresponding to the word. Furthermore, as described herein, output data 608 may represent one or more letters, numbers, characters, punctuation marks, and / or any other type of identifier. For example, lowercase indicators 612, 618, and 620 may contain a first identifier, while uppercase indicators 610, 614, and 616 may contain a second identifier.

[0068] Return to reference Figure 5 As an example, process 500 may also include applying embedding 114 to one or more punctuation models 510. As described herein, in some examples, punctuation model 510 may include, but is not limited to: one or more machine learning models, one or more neural networks, one or more layers of one or more models (e.g., one or more connection layers, one or more output layers, etc.), one or more heads of one or more models, software, hardware, and / or any other type of processing components configured to perform at least a portion of the process described herein. For example, at least based on processing of embedding 114, process 500 may include punctuation model 510 generating and / or outputting speech-associated output data 512.

[0069] As shown in the figure, in some examples, output data 512 may represent at least one or more word indicators 514 associated with speech and / or one or more punctuation indicators 516 associated with speech. As described herein, in some examples, word indicators 514 may indicate that a word does not contain any punctuation, while punctuation indicators 516 may indicate that a word does contain punctuation and / or indicate the type of punctuation. As described herein, the type of punctuation may include, but is not limited to, periods, exclamation marks, question marks, commas, and / or any other type of punctuation.

[0070] Furthermore, in some examples, word indicator 514 and / or punctuation indicator 516 can be associated with various probabilities. For example, for a word, word indicator 514 can indicate a first probability that the word is not associated with any punctuation mark; first punctuation indicator 516 can indicate a second probability that the word is associated with a first type of punctuation mark (e.g., a period); second punctuation indicator 516 can indicate a third probability that the word is associated with a second type of punctuation mark (e.g., an exclamation mark); third punctuation indicator 516 can indicate a fourth probability that the word is associated with a third type of punctuation mark (e.g., a question mark); fourth punctuation indicator 516 can indicate a fifth probability that the word is associated with a fourth type of punctuation mark (e.g., a comma), and so on. In some examples, the sum of probabilities can be a specific probability, such as 100% (and / or any other probability).

[0071] For example, Figures 7A-7B Examples of performing speech-associated punctuation detection according to some embodiments of this disclosure are shown. Figure 7A As shown in the example, from Figure 2 The example data 202 (e.g., the embedding) can also be applied to the punctuation model 510. Therefore, the punctuation model 510 can perform one or more of the processes described herein to generate output data 702 associated with data 202 (which may resemble and / or represent output data 512). For example, as shown, output data 702 may represent the probabilities 704(1)-704(6) that six words are not associated with any punctuation, the probabilities 706(1)-706(6) that six words are associated with punctuation of the first type, and so on, up to the probabilities 708(1)-708(6) that six words are associated with punctuation of the last type.

[0072] Although Figure 7A The example shows the determination of output data 702 (e.g., probabilities) associated with individual words in the text, but in other examples, a similar process can be used to determine the probabilities associated with individual tokens corresponding to the text. For example, probability 704(1) may be associated with a token that does not correspond to a punctuation mark, probability 706(1) may be associated with a token that corresponds to a punctuation mark of the first type, and so on, until probability 708(1) may be associated with a token that corresponds to a punctuation mark of the last type.

[0073] Return to reference Figure 5For a first example, process 500 can include detection component 126 further processing output data 512 to detect an EOS position and / or an EOU position associated with the speech. For the first example, when the probability associated with the end-of-sentence punctuation (e.g., period, question mark, exclamation point, etc.) contains the highest probability and / or satisfies a threshold probability, detection component 126 can determine that the word is likely associated with an EOS position and / or an EOU position. For a second example, when the probability associated with the in-sentence punctuation (e.g., comma, etc.) contains the highest probability and / or satisfies a threshold probability, detection component 126 can determine that the word is not likely associated with an EOS position and / or an EOU position. Still, for a third example, when the probability associated with no punctuation contains the highest probability and / or satisfies a threshold probability, detection component 126 can determine that the word is not likely associated with an EOS position and / or an EOU position. While these are merely some example techniques by which detection component 126 can utilize punctuation detection to further detect EOS and / or EOU positions, in other examples, detection component 126 can use other and / or alternative techniques.

[0074] For example, Figure 7B Another example of performing punctuation detection associated with speech is shown in accordance with some embodiments of the present disclosure. As shown, detection component 126 can use output data 702 to generate output data 710 representing indicators of whether words contain punctuation. For example, detection component 126 can determine that a first word does not contain punctuation 712 based at least on probability 704(1) containing the highest probability and / or satisfying a threshold probability. Further, detection component 126 can determine that a second word contains a first type of punctuation (e.g., a comma 714) based at least on probability 706(2) containing the highest probability and / or satisfying a threshold probability. Detection component 126 can then perform a similar process to determine that a third word includes a period 716, a fourth word does not contain punctuation 718, a fifth word does not include punctuation 720, and a sixth word includes a question mark 722. Detection component 126 can then use output data 710 in generating output data 302 in accordance with some embodiments of the present disclosure. Figure 3 For example,

[0075] While Figure 7BThe example illustrates the generation of output data 710 for each word, but in other examples, the detection component 126 may generate similar output data 710 for each token associated with the text corresponding to the word. Furthermore, as described herein, the output data 710 may represent one or more letters, numbers, characters, punctuation marks, and / or any other type of identifier. For example, punctuation marks 712, 718, and 720 may contain a first identifier, a comma 714 may contain a second identifier, a period 716 may contain a third identifier, and a question mark 722 may contain a fourth identifier.

[0076] Return to reference Figure 5 The example, although Figure 5 The example shows that recognition model 104, encoder 112, detection model 116, detection component 126, case model 502, and punctuation model 510 are separate from each other. However, in other examples, recognition model 104, encoder 112, detection model 116, detection component 126, case model 502, and / or punctuation model 510 can be combined into one or more machine learning models. For example, a model may include at least layers of encoder 112, layers of detection model 116, layers of case model 502, layers of punctuation model 510, and / or layers of detection component 126.

[0077] Figure 8 A data flow diagram illustrating a process 800 for training detection model 116, case model 502, and / or punctuation model 510 to perform interruption detection is shown according to some embodiments of the present disclosure. As shown, detection model 116, case model 502, and / or punctuation model 510 can be trained using training text data 802. In some examples, training text data 802 may include text instances associated with utterances, which may resemble text data 106 and / or training text data 402 and / or include text data 106 and / or training text data 402. Training text data 802 may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data such as audio data), machine-automated generated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., taggers or annotation experts define the placement of labels), and / or a combination thereof.

[0078] The detection model 116, the case model 502, and / or the punctuation model 510 can also be trained using respective ground truth data 804. The ground truth data 804 can include annotations, labels, masks, and the like. For example, as shown, the ground truth data 804 can include at least one or more detection indicators 806, one or more case indicators 808, and / or one or more punctuation indicators 810. The ground truth data 804 can be synthetically generated (e.g., generated by a computer model or render graph), realistically generated (e.g., designed and generated from real-world data), machine-automated generated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated generated (e.g., a labeler or annotation expert defines the location of labels), and / or a combination thereof. In some examples, there can be a corresponding ground truth data 804 for each instance of the training text data 802.

[0079] In some examples, the detection indicators 806 can indicate one or more normal words, one or more EOS words, and one or more EOU words, such as similar to the word indicators 406, the EOS indicators 408, and the EOU indicators 410 in the example of Figure 4 In addition, in some examples, the case indicators 808 can indicate whether one or more words are lower case and / or whether one or more words are upper case, such as similar to the lower case indicators 506 and / or the upper case indicators 508 in the example of Figure 5 In addition, in some examples, the punctuation indicators 810 can indicate whether one or more words are associated with punctuation and / or indicate one or more types of punctuation with which one or more words are associated, such as similar to the word indicators 514 and / or the punctuation indicators 516 in the example of Figure 5

[0080] As Figure 8 ​Further shown, one or more training engines 812 can use one or more loss functions that measure a loss (e.g., an error) of the output 814(1)-814(3) compared to the ground truth data 804. In some examples, the output 814(1) can be similar to and / or contain the output data 118, the output 814(2) can be similar to and / or contain the output data 504, and / or the output 814(3) can be similar to and / or contain the output data 512. Any type of loss function can be used, such as a cross-entropy loss, a mean squared error, a mean absolute error, a mean bias error, and / or other types of loss functions. In some examples, different outputs 814(1)-814(3) can have different loss functions. For example, a first loss function can be used and / or a first loss can be determined based at least on comparing the output 814(1) to the detection indicator 806, a second loss function can be used and / or a second loss can be determined based at least on comparing the output 814(2) to the case indicator 808, and / or a third loss function can be used and / or a third loss can be determined based at least on comparing the output 814(3) to the punctuation indicator 810.

[0081] In certain examples, in determining different losses, the loss functions and / or losses can be combined to form a total loss (where one or more losses can be weighted) and the total loss can be used to train the detection model 116, the case model 502, and / or the punctuation model 510 (e.g., update parameters of the detection model 116, the case model 502, and / or the punctuation model 510). However, in other examples, the detection model 116 can be trained using the first loss function and / or the first loss, the case model 502 can be trained using the second loss function and / or the second loss, and the punctuation model 510 can be trained using the third loss function and / or the third loss. In any examples, backpropagation calculations can be performed to recursively compute gradients of the loss functions with respect to the training parameters. In some examples, weights and biases of the detection model 116, the case model 502, and / or the punctuation model 510 can be used to compute these gradients.

[0082] Figure 9An example illustrating a third process 900 for performing speech disruption detection using fusion in accordance with some embodiments of the present disclosure is shown. As shown, in addition to process 100 and / or process 500, process 900 can include processing at least a portion of audio data 102 using one or more encoders 902, such as one or more audio encoders, configured to generate one or more embeddings 904 representative of audio data 102. As described herein, in some examples, encoders 902 can include any type of audio encoder capable of generating embeddings 904 based at least on processing of audio data 102.

[0083] As Figure 9 further illustrated by the example, process 900 can include processing at least a portion of embeddings 114 and / or at least a portion of embeddings 904 using one or more fusion components 906. Based at least on this processing, process 900 can include fusion components 906 generating one or more additional embeddings 908 for application to detection model 116, case model 502, and / or punctuation model 510. For example, fusion components 906 can generate embeddings 908 by combining, fusing, mixing, and / or performing any other type of process related to embeddings 114 and embeddings 904.

[0084] While Figure 9 the example illustrates adding encoders 902 and fusion components 906, in other examples, encoders 902 and / or fusion components 906 can be added to the example of process 100. For example, process 100 can use encoders 902 and fusion components 906, where embeddings 908 are subsequently applied only to detection model 116. Further, in some examples, process 900 can not include one or more of case model 502 and / or punctuation model 510. Figure 1

[0085] As described herein, in some examples, at least one of recognition model 104, language model 108, encoder 112, detection model 116, detection component 126, case model 502, punctuation model 510, encoders 902, and / or fusion components 906 can be stored on and / or executed by one or more computing devices. For example, at least one of recognition model 104, language model 108, encoder 112, detection model 116, detection component 126, case model 502, punctuation model 510, encoders 904, and / or fusion components 908 can be stored in one or more memories and / or executed by one or more processors of computing device 1500 and / or example data center 1600, which will be described in greater detail herein.

[0086] For example, Figure 10 ​An example of a system 1000 that can perform one or more processes described herein is shown in accordance with some embodiments of the present disclosure. As shown, the system 1000 (which can represent and / or include the example computing device 1500 and / or the example data center 1600) can include one or more processors 1102 (which can be similar to and / or include one or more central processing units 1506 and / or one or more graphics processing units 1508) and a memory 1006 (which can be similar to and / or include the memory 1504). For example, the memory 1006 can store the recognition model 104, the language model 108, the encoder 112, the detection model 116, the detection component 126, the capitalization model 502, the punctuation model 510, the encoder 902, and / or the fusion component 906. Additionally, the processor 1004 can execute the recognition model 104, the language model 108, the encoder 112, the detection model 116, the detection component 126, the capitalization model 502, the punctuation model 510, the encoder 902, and / or the fusion component 906 to perform one or more processes described herein.

[0087] Further, as shown by the example of Figure 10 The system 1000 can receive audio data 102 from one or more client devices 1008 (which can also be similar to and / or include the example computing device 1500) and / or send output data 110 to the client devices 1008, as shown by the example of Figure 10 For example, the client devices 1008 can generate the audio data 102 using one or more input devices, such as one or more microphones. The client devices 1008 can also include one or more output devices, such as one or more speakers, to output 1010 sound associated with the output data 110. For example, in some examples, the audio data 102 can represent a query, and the output data 110 can represent a response to the query. Although the example of

[0088] Reference is now made to Figures 11-13Each of the blocks of the methods 1100, 1200, and 1300 described herein can include a computer process that can be performed by a hardware processor, firmware processor, and / or hardware functioning as a stored program control processor. The various functions performed in methods 1100, 1200, and 1300 can be implemented using any combination of hardware, firmware, and / or software. For instance, various functions can be implemented with any combination of: a processor executing stored instructions; software modules running on the processor and / or firmware; firmware stored in memory; hardware logic / circuitry; etc. The methods 1100, 1200, and 1300 can also be embodied as computer-usable instructions stored on computer storage media (e.g., both non-transitory computer storage media and / or transitory computer storage media, e.g., electrical, optical, electromagnetic, infrared, and so forth, including human-readable or human- understandable and / or machine-readable or machine- understandable media). The computer-usable instructions can be executed by one or more processors to cause the processors to perform at least some of the processes described herein. The computer-usable instructions can be stored on computer storage media, which can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code in the form of computer-usable instructions or data structures and that can be accessed by a computer. Also, functional computer programs, software, routines, or instructions can be embedded in Figure 1 The methods 1100, 1200, and 1300 are described herein, by way of example, with reference to the systems illustrated in FIGS. 1-9. However, the methods 1100, 1200, and 1300 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0089] Figure 11 A flowchart illustrating a method 1100 for determining at least sentence endings and utterance endings associated with tokens, in accordance with some embodiments of the present disclosure is shown. The method 1100 can include, at block B1102, generating, based at least on audio data representing an utterance, textual data corresponding to the utterance. For example, the recognition model 104 can process at least a portion of the audio data 102 representing an utterance. Based at least on this processing, the recognition model 104 can generate textual data 106 associated with the utterance. As described herein, in some examples, the textual data 106 can correspond to one or more tokens, such as one or more text tokens corresponding to text associated with the utterance. Further, in some examples, the encoder 112 can then generate embeddings 114 associated with the textual data 106.

[0090] The method 1100 can include, at block B1104, generating, using one or more first models and based at least on the textual data, output data indicating whether each token corresponding to the textual data is associated with a sentence ending in the utterance or with an ending of the utterance. For example, the detection model 116 can generate, based at least on the textual data 106 (and / or the embeddings 114), output data 118. As described herein, in some examples, the output data 118 can represent at least word indicators 120, EOS indicators 122, and / or EOU indicators 124 associated with each token. For example, for a token, the word indicators 120 can indicate whether the token is associated with a normal word, the EOS indicators 122 can indicate whether the token is associated with an EOS position, and the EOU indicators 124 can indicate whether the token is associated with an EOU position.

[0091] The method 1100 can include, at block B1106, determining, based at least on the output data, a first location in the textual data associated with an end of sentence and a second location in the textual data associated with an end of utterance. For example, the detection component 126 can determine, based at least on the output data 118, a first location in the textual data 106 associated with an EOS and a second location in the textual data 106 associated with an EOU. As described herein, in some examples, the detection component 126 can determine that the first location is associated with an EOS based at least on the probability associated with the EOS indicator 122 including a highest probability and / or satisfying a threshold probability. Further, in some examples, the detection component 126 can determine that the second location is associated with an EOU based at least on the probability associated with the EOU indicator 124 including a highest probability and / or satisfying a threshold probability.

[0092] The method 1100 can include, at block B1108, processing, using one or more second models and based at least on the first location and the second location, a first portion of the textual data corresponding to a sentence of the utterance before processing a second portion of the textual data corresponding to a remaining portion of the utterance. For example, the language model 108 can process a first portion of the textual data 106 based at least on the EOS location being detected. Next, the language model 108 can process a second portion of the textual data 106 based at least on the EOU location being detected. As described herein, in some examples, the second portion of the textual data 106 can represent the entire utterance and / or a portion of the utterance after the sentence. By detecting the EOS of each sentence, the data corresponding to the sentence can be sent for processing before waiting for the entire utterance to be identified for processing, which reduces latency (relative to previous methods, i.e., waiting for the entire utterance to be identified before sending for processing by downstream models) and, in some cases, allows the model to have context for processing the entire utterance based on having already processed one or more sentences of the utterance identified using the EOS detection.

[0093] Figure 12 A flow diagram illustrating a method 1200 for performing speech break detection is shown, in accordance with some embodiments of the present disclosure. The method 1200 can include, at block B1202, generating, based at least on audio data representing speech, textual data associated with one or more words corresponding to the speech. For example, the recognition model 104 can process at least a portion of the audio data 102 representing speech. Based at least on the processing, the recognition model 104 can generate textual data 106 associated with text corresponding to the speech. As described herein, in some examples, the textual data 106 can represent one or more tokens associated with the speech, such as one or more text tokens. Further, in some examples, the encoder 112 can then generate an embedding 114 associated with the textual data 106.

[0094] The method 1200 can include generating, using the one or more models and based at least on the text data, output data representing that one or more words are associated with an end-of-sentence or an end-of-utterance, at block B1204. For example, the detection model 116 can generate the output data 118 based at least on the text data 106 (and / or the embedding 114). As described herein, in some examples, the output data 118 can represent at least the word indicators 120, the EOS indicator 122, and / or the EOU indicator 124 associated with the words in the speech. Further, in some examples, the word indicators 120, the EOS indicator 122, and / or the EOU indicator 124 can be associated with one or more probabilities.

[0095] The method 1200 can include determining, based at least on the output data, a position in the one or more words that is associated with at least one of an end-of-sentence or an end-of-utterance, at block B1206. For example, the detection component 126 can use the output data 118 to determine a position in the words that is associated with the EOS or the EOU. As described herein, in some examples, the detection component 126 can determine that the position is associated with the EOS based at least on the probability associated with the EOS indicator 122 including a highest probability and / or satisfying a threshold probability. Further, in some examples, the detection component 126 can determine that the position is associated with the EOU based at least on the probability associated with the EOU indicator 124 including a highest probability and / or satisfying a threshold probability. Further, in some examples, the detection component 126 can include at least a portion of the detection model 116.

[0096] The method 1200 can include generating, using the one or more language models and based at least on at least a portion of the text data associated with the position, output associated with the speech, at block B1208. For example, at least a portion of the text data 106 can be processed by the language model 108, where the at least a portion of the text data 106 is associated with the position. For example, the at least a portion of the text data 106 can be associated with one or more words that begin at a starting word and end at the position within the word. Further, based at least on the processing of the at least a portion of the text data 106, the language model 108 can generate and / or output the output data 110 associated with the speech.

[0097] Figure 13A flow diagram illustrating a method 1300 for determining an end-of-sentence and an end-of-utterance associated with speech, in accordance with some embodiments of the present disclosure, is shown. The method 1300 can include, at block B1302, generating text data associated with one or more words corresponding to the speech based at least on audio data representative of the speech. For example, the recognition model 104 can process at least a portion of the audio data 102 representative of the speech. Based at least on the processing, the recognition model 104 can generate text data 106 associated with text corresponding to the speech. As described herein, in some examples, the text data 106 can be representative of one or more tokens associated with the speech, such as one or more text tokens. Further, in some examples, the encoder 112 can then generate an embedding 114 associated with the text data 106.

[0098] The method 1300 can include, at block B1304, generating output data representative of one or more first probabilities that one or more words are associated with an end of a sentence and one or more second probabilities that one or more words are associated with an end of an utterance based at least on the text data using one or more models. For example, the detection model 116 can generate output data 118 based at least on the text data 106 (and / or the embedding 114), where the output data 118 is representative of one or more first probabilities associated with the EOS indicator 122 and one or more second probabilities associated with the EOU indicator 124.

[0099] The method 1300 can include, at block B1306, determining at least one of an end-of-sentence or an end-of-utterance associated with the speech based at least on the output data. For example, the detection component 126 can use the one or more first probabilities and the one or more second probabilities to determine the EOS and / or the EOU. As described herein, in some examples, the detection component 126 can determine the EOS based at least on the first probability associated with the EOS indicator 122 including a highest probability and / or satisfying a threshold probability. Further, in some examples, the detection component 126 can determine the EOU based at least on the second probability associated with the EOU indicator 124 including a highest probability and / or satisfying a threshold probability. Further, in some examples, the detection component 126 can include at least a portion of the detection model 116.

[0100] Example language model

[0101] In at least some embodiments, language models, such as large language models (LLMs) and / or other types of generative artificial intelligence (AI) can be implemented. These models can be capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer aided design (CAD) assets, metaverse and / or omniverse file information (e.g., USD format), etc. based on context provided in an input prompt or query. In embodiments, these language models can be considered “large” in that these models are trained on massive datasets and have architectures that contain a large number of learnable network parameters (weights and biases), such as millions or billions of parameters. LLMs / VLMs / etc. can be implemented for summarizing textual data, analyzing and extracting insights in data (e.g., text, images, videos, etc.), and generating new text / images / videos / etc. in a user-specified style, tone, or format. In embodiments, the LLMs of the present disclosure can be specialized for text processing, while in other embodiments, multi-modal LLMs can be implemented to accept, understand, and / or generate text as well as other types of content, such as images, audio, and / or video. For example, a visual language model (VLM), or more generally a multi-modal language model, can be implemented to accept images, videos, audio, text, 3D designs (e.g., CAD), and / or other input data types, and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0102] Various types of LLM / VLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented that use different techniques to understand and generate output— e.g., text, audio, video, images, etc. In some embodiments, LLM architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can be used, while in other embodiments, transformer architectures (such as those that rely on self-attention mechanisms) can be used to understand and recognize relationships between words or tokens. The language models of the present disclosure can include encoder and / or decoder blocks. For example, discriminative or encoder-only LLMs, such as BERT (Bidirectional Encoder Representations from Transformers), can be implemented for tasks involving language understanding, such as classification, sentiment analysis, question answering, and named entity recognition. For another example, generative or decoder-only LLMs, such as GPT (Generative Pretrained Transformer), can be implemented for tasks involving language and content generation, such as text completion, story generation, and conversation generation. LLMs that contain both encoder and decoder components, such as T5 (Text-to-Text Transformer), can be implemented for understanding and generating content, such as for translation and summarization. These examples are not intended to be limiting, and any architecture type (including but not limited to the architecture types described herein) can be implemented depending on the specific embodiment and the tasks the model is performing.

[0103] In various embodiments, LLMs / VLMs, etc. can be trained using unsupervised learning, where the LLM learns patterns from large amounts of unlabeled text / audio / video / image, etc. data. Due to the large amount of training, in embodiments, the model can not need to be trained for a particular task or a particular domain. LLMs that have been extensively pre-trained on large amounts of unlabeled text data can be referred to as base models, and can be good at a variety of tasks, such as question answering, summarization, filling in missing information, and translation. Certain LLMs can be customized for particular use cases using techniques such as quick adaptation, fine-tuning, retrieval-augmented generation (RAG), etc., add adapters (e.g., customized neural networks and / or neural network layers that adjust or adapt prompts or tokens to bias the language model towards a particular task or domain), and / or use optimization models for other fine-tuning or customization techniques for particular tasks and / or particular domains.

[0104] In some embodiments, LLMs / VLMs, etc. of the present disclosure can be implemented using various model alignment techniques. For example, in some embodiments, safeguards can be implemented to identify inappropriate or unintended inputs (e.g., prompts) and / or outputs of the model. In some non-limiting embodiments, the implemented safeguards can be similar to those described in U.S. Patent Application No. 18,304,341, filed April 20, 2023, the contents of which are incorporated by reference herein in their entirety. In some embodiments, one or more additional models (or layers thereof) can be implemented to identify issues with inputs and / or outputs of the model. For example, these “safety” models can be trained to identify inputs and / or outputs that are “safe” or otherwise good or desirable for a particular application / implementation, and / or inputs and / or outputs that are “unsafe” or otherwise undesirable. Thus, LLMs / VLMs, etc. of the present disclosure are less likely to output language / text / audio, etc. that can be offensive, vulgar, inappropriate, unsafe, out of scope, and / or otherwise undesirable for a particular application / implementation.

[0105] In some embodiments, the LLM / VLM / etc. can be configured to or have access to or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations for which the model is not ideally suited, the model can have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is if at least a portion of the response requires mathematical calculations, the model can access one or more mathematical plugins or APIs to help solve the problem, which can then be used in the output of the model from the plugin and / or API. This process can repeat (e.g., recursively) for any number of iterations and using any number of plugins and / or APIs until a response to the input prompt that solves each query / question / request / process / operation / etc. can be generated. Thus, the model can rely not only on its own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (e.g., APIs, plugins, etc.).

[0106] Figure 14A is a block diagram of an example generative language model system 1400 suitable for implementing at least some embodiments of the present disclosure. In Figure 14A In the example shown, the generative language model system 1400 includes a retrieval-augmented generation (RAG) component 1492, an input processor 1405, a tokenizer 1410, an embedding component 1420, plugins / APIs 1495, and a generative language model (LM) 1430 (which can include an LLM, a VLM, a multimodal LM, etc.).

[0107] At a high level, input processor 1405 can receive input 1401 containing text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasonic, etc.), 3D design data, CAD data, Universal Scene Description (USD) data, etc.), depending on the architecture of the generative LM 1430. In some embodiments, input 1401 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 1401 can include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., table format, JSON, or XML). In some implementations where the generative LM 1430 is capable of processing multi-modal input, input unit 1401 can combine text with image data, audio data, and / or other types of input data such as, but not limited to, the data described herein. Taking the example of raw input text, input processor 1405 can prepare the raw input text in various ways. For example, input processor 1405 can perform various types of text cleaning to remove noise (e.g., special characters, punctuation, HTML tags, stop words) from the relevant textual content. In examples involving stop words (commonly used words that tend to have less semantic meaning), input processor 1405 can remove stop words to reduce noise and allow the generative LM 1430 to focus on more meaningful content. Input processor 1405 can apply text normalization, e.g., by converting all characters to lowercase, removing diacritics, and / or handling special cases such as acronyms or abbreviations to ensure consistency. These are just a few examples, and other types of input processing can also be applied.

[0108] In some embodiments, the RAG component 1492 can be used to retrieve additional information to be used as part of the input 1401 or prompt. For example, in some embodiments, the input 1401 can be generated using a query or model input (e.g., a question, request, etc.) in addition to data retrieved using the RAG component 1492. In some embodiments, the input processor 1405 can analyze the input 1401 and communicate with the RAG component 1492 (or, in embodiments, the RAG component 1492 can be part of the input processor 1405) in order to identify relevant text and / or other data to provide to the generative LM 1430 as additional context or source of information from which to typically identify a response, answer, or output 1490. For example, when the input indicates that the user is interested in the expected tire pressure for a particular make and model vehicle, the RAG component 1492 can retrieve tire pressure information or text corresponding thereto (e.g., using a vector search in an embedding space) from a digital (embedded) version of the user manual for that particular vehicle make and model. Similarly, when the user revisits a chatbot related to a particular product or service, the RAG component 1492 can retrieve a previously stored conversation history (or at least an abridged version thereof) and provide the previous conversation history along with the current inquiry / request as part of the input 1401 to the generative LM 1430.

[0109] The tokenizer 1410 can segment (e.g., processed) text into smaller units (tokens) for subsequent analysis and processing. Depending on the implementation, tokens can represent individual words, subwords, characters, etc. Word-based tokenization divides text into individual words, treating each word as a separate token. Subword tokenization breaks down words into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 1430 to understand morphological variations and more effectively process out-of-vocabulary words. Character-based tokenization represents each character as a separate token, enabling the generative LM 1430 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or characteristics of the training dataset. Accordingly, the tokenizer 1410 can transform (e.g., processed) text into a structured format according to a tokenization scheme implemented in a particular embodiment.

[0110] The embedding component 1420 can transform discrete tokens into a semantically meaningful (e.g., dense, continuous vector) representation using any known embedding technique. For example, the embedding component 1420 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0111] In some implementations in which the input 1401 includes image data, the input processor 1401 can resize the image data to a standard size compatible with the format of the respective input channel and / or can normalize pixel values to a common range (e.g., 0 to 1) to ensure consistent representation, and the embedding component 1420 can encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations in which the input 1401 includes audio data, the input processor 1401 can resample audio files to a consistent sampling rate for uniform processing, and the embedding component 1420 can extract and encode audio features using any known technique, e.g., in the form of a spectrogram (e.g., a mel-spectrogram). In some implementations in which the input 1401 includes video data, the input processor 1401 can extract frames or apply resizing to extracted frames, and the embedding component 1420 can extract features such as optical flow embeddings or video embeddings and / or can encode temporal information or sequences of frames. In some implementations in which the input 1401 includes multi-modal data, the embedding component 1420 can fuse representations of different types of data (e.g., text, images, audio) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion, and / or others.

[0112] The generative LM 1430 and / or other components of the generative LLM system 1400 can use different types of neural network architectures depending on the implementation. For example, a transformer-based architecture (such as used in GPT and the like) can be implemented, and it can include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feed-forward network that processes the output of the self-attention layer, which applies a non-linear transformation to the input representation and extracts higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multi-modal), RNNs, LSTMs, fusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectures adversarial networks such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning, and the like. Thus, depending on the implementation and architecture, the embedding component 1420 can apply the encoded representation of the input 1401 to the generative LM 1430, and the generative LM 1430 can process the encoded representation of the input 1401 to generate an output 1490, which can include response text and / or other types of data.

[0113] As described herein, in some embodiments, generative LM 1430 can be configured to access or use (or be able to access or use) plugins / APIs 1495 (which can include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations that are not ideally suited for generative LM 1430, the model can have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 1492) for accessing one or more plugins / APIs 1495 (e.g., third-party plugins) to assist in processing the current input. In such examples, when at least a portion of the prompt is related to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs), send at least a portion of the prompt related to the particular plugin / API 1495 to the plugin / API 1495, the plugin / API 1495 can process that information and return an answer to generative LM 1430, which can use that response to generate output 1490. This process can repeat (e.g., recursively) for any number of iterations and with any number of plugins / APIs 1495 until output 1490 can be generated that addresses each query / question / request / process / operation / etc. from input 1401. Thus, the model can rely not only on its own knowledge obtained from training on large datasets and / or from data retrieved using RAG component 1492, but also on the specialized knowledge or optimized properties of one or more external resources (e.g., plugins / APIs 1495).

[0114] Figure 14B is a block diagram of an example implementation in which generative LM 1430 includes a transformer encoder-decoder. For example, assume that input text (e.g., “Who discovered gravity”) is tokenized (e.g., by tokenizer 1410 of Figure 14A Figure 14A is tokenized (e.g., by tokenizer 1410 of

[0115] ​In example implementations, the encoder 1435 forms an encoder stack in which each encoder includes a self-attention layer and a feed-forward network. In an example transducer architecture, each token (e.g., word) flows through a separate path. Thus, each encoder can accept a sequence of vectors, pass each vector through a self-attention layer, then through a feed-forward network, and then pass up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token, a self-attention score for a pair of tokens can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention in which the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be concatenated to generate a context vector that encodes the input. The attention projection layer 1440 can transform the context vector into an attention vector (keys and values) for the decoder 1445.

[0116] In example implementations, the decoder 1445 forms a decoder stack in which each decoder includes a self-attention layer, an encoder-decoder self-attention layer that attends to relevant portions of the input sequence using attention vectors (keys and values) from the encoder, and a feed-forward network. As with the encoder 1435, in an example transducer architecture, each token (e.g., word) flows through a separate path in the decoder 1445. During a first pass, the decoder 1445, the classifier 1450, and the generation mechanism 1455 can generate a first token, and the generation mechanism 1455 can apply the generated token as input during a second pass. This process can be repeated iteratively, generating tokens (e.g., words) one at a time and adding them to the output of the previous pass, and applying the token embedding of the composite sequence with positional encodings as input to the decoder 1445 in subsequent passes, iteratively one token at a time (known as autoregressive), until a symbol or token is predicted that indicates the end of the response. In each decoder, the self-attention layer is typically restricted to attending only to previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In example implementations, the operation of the encoder-decoder attention layer is similar to that of the (e.g., multi-head) self-attention in the encoder 1435, except that it creates its queries from the layer below it and takes its keys and values (e.g., matrices) from the output of the encoder 1435.

[0117] Accordingly, the decoders 1445 can output some decoded (e.g., vector) representation of the input applied during a particular pass. The classifier 1450 can include a multi-class classifier that includes one or more neural network layers that project the decoded (e.g., vector) representation to respective dimensions (e.g., one dimension for each supported word or token in the output vocabulary) and a softmax operation that converts logits to probabilities. Accordingly, the generation mechanism 1455 can select or sample a word or token based on the respective predicted probabilities (e.g., select the word with the highest predicted probability) and append it to the output of the previous pass, generating each word or token in order. The generation mechanism 1455 can repeat the process, triggering successive decoder inputs and respective predictions, until a symbol or token is selected or sampled that represents the end of the response, at which point the generation mechanism 1455 can output the generated response.

[0118] Figure 14C is a block diagram of an example implementation in which the generative LM 1430 includes a decoder-only transducer architecture. For example, Figure 14C The decoders 1460 can operate similarly to the decoders 1445 of Figure 14B except that Figure 14C Each decoder 1460 of the decoder-only transducer architecture omits the encoder-decoder self-attention layer (as there is no encoder in this implementation). Accordingly, the decoders 1460 can form a stack of decoders, where each decoder includes a self-attention layer and a feedforward network. Further, a symbol or token representing the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, rather than encoding the input sequence, and the resulting sequence (e.g., respective embeddings with position encodings) can be applied to the decoders 1460. As with the decoders 1445 of Figure 14B Each token (e.g., word) can flow through a separate path in the decoders 1460, and the decoders 1460, classifier 1465, and generation mechanism 1470 can use autoregression to generate the tokens one at a time in order until a symbol or token is predicted that represents the end of the response. The classifier 1465 and generation mechanism 1470 can operate similarly to the classifier 1450 and generation mechanism 1455 of Figure 14B where the generation mechanism 1470 selects or samples each successive output token based on the respective predicted probabilities and appends it to the output of the previous pass, generating each token in order until a symbol or token is selected or sampled that represents the end of the response. These and other architectures described herein are by way of example only, and other suitable architectures can be implemented within the scope of the present disclosure.

[0119] Example Computing Device

[0120] Figure 15is a block diagram of an example computing device 1400 suitable for implementing some embodiments of the present disclosure. The computing device 1500 can include an interconnection system 1502 coupling the following components: a memory 1504, one or more central processing units (CPU) 1506, one or more graphics processing units (GPU) 1508, a communication interface 1510, an input / output (I / O) port 1512, an input / output component 1514, a power supply 1516, one or more presentation components 1518 (e.g., one or more displays), and one or more logic units 1520. In at least one embodiment, one or more computing devices 1500 can include one or more virtual machines (VMs), and / or any component thereof can include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more of GPUs 1508 can include one or more vGPUs, one or more of CPUs 1506 can include one or more vCPUs, and / or one or more of logic units 1520 can include one or more virtual logic units. As such, one or more computing devices 1500 can include discrete components (e.g., a full GPU dedicated to computing device 1500), virtual components (e.g., a portion of a GPU dedicated to computing device 1500), or a combination thereof.

[0121] Although Figure 15 various blocks of are shown as connected through an interconnection system 1502 using lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1518, such as a display device, can be considered an I / O component 1514 (e.g., if the display is a touchscreen). As another example, a CPU 1506 and / or GPU 1508 can include memory (e.g., memory 1504 can represent a storage device in addition to memory of GPU 1508, CPU 1506, and / or other components). Thus, Figure 15 computing devices of are merely illustrative. There is no distinction, in terms of Figure 15 scope, between the computing devices of

[0122] The interconnection system 1502 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1502 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1506 can be directly connected to the memory 1504. Further, the CPU 1506 can be directly connected to the GPU 1508. Where there are direct or point-to-point connections between components, the interconnection system 1502 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1500.

[0123] The memory 1504 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1500. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0124] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, the memory 1504 can store computer readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1500. Computer storage media, as used herein, does not include signals per se.

[0125] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above can also be included within the scope of computer readable media.

[0126] The CPUs 1506 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1500 to perform one or more of the methods and / or processes described herein. The CPUs 1506 can each include one or more cores capable of handling numerous software threads concurrently (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). The CPUs 1506 can include any type of processors, and can include different types of processors depending on the type of computing device 1500 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1500, the processors can be Advanced RISC Machines (ARM) processors implemented using Reduced Instruction Set Computing (RISC) or x86 processors implemented using Complex Instruction Set Computing (CISC). The computing device 1500 can include one or more CPUs 1506 in addition to, or as an alternative to, one or more microprocessors or supplemental co-processors such as math co-processors.

[0127] One or more GPUs 1508, in addition to or in place of one or more CPUs 1506, can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. One or more of GPUs 1508 can be integrated GPUs (e.g., with one or more of CPUs 1506) and / or one or more of GPUs 1508 can be discrete GPUs. In embodiments, one or more of GPUs 1508 can be a co-processor of one or more of CPUs 1506. GPUs 1508 can be used by computing device 1500 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, GPUs 1508 can be used for general-purpose computing on GPUs (GPGPU). GPUs 1508 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. GPUs 1508 can generate pixel data for output images in response to rendering commands (e.g., received from CPUs 1506 via a host interface). GPUs 1508 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory can be included as part of memory 1504. GPUs 1508 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 1508 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a simulated image). Each GPU can include its own memory or can share memory with other GPUs.

[0128] In addition to or in lieu of CPU(s) 1506 and / or GPU(s) 1508, logic units 1520 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPU(s) 1506, one or more GPU(s) 1508, and / or one or more logic units 1520 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic units 1520 can be part of one or more of CPU(s) 1506 and / or GPU(s) 1508 and / or one or more of logic units 1520 can be discrete components or otherwise external to CPU(s) 1506 and / or GPU(s) 1508. In embodiments, one or more of logic units 1520 can be a co-processor of one or more of CPU(s) 1506 and / or GPU(s) 1508.

[0129] Examples of logic units 1520 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multi-processor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), a programmable vision accelerator (PVA) which can include one or more direct memory access (DMA) systems, one or more visual or vector processing units (VPUs), one or more pixel processing engines (PPEs), etc., a visual processing unit (VPU), an optical flow accelerator (OFA), a field programmable gate array (FPGA), a neuromorphic chip, a quantum processing unit (QPU), an associative processing unit (APU), an arithmetic logic unit (ALU), an application-specific integrated circuit (ASIC), a floating-point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) element, etc.

[0130] The communication interface 1510 can include one or more receivers, transmitters, and / or transceivers that enable the computing device 1500 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 1510 can include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1520 and / or the communication interface 1510 can include one or more data processing units (DPUs) to transfer data received over networks and / or through the interconnect system 1502 directly to the one or more GPUs 1508 (e.g., memory of the one or more GPUs 1508).

[0131] The I / O ports 1512 can enable the computing device 1500 to be logically coupled to other devices including I / O components 1514, one or more presentation components 1518, and / or other components, some of which can be built into (e.g., integrated with) the computing device 1500. Illustrative I / O components 1514 include a microphone, mouse, keyboard, joystick, game pad, game controller, dish satellite antenna, scanner, printer, wireless device, etc. The I / O components 1514 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, inputs are transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, pen / mouse recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1500. The computing device 1500 can include a depth camera, such as a stereoscopic camera system, infrared camera system, RGB camera system, touch screen technology, and combinations of these, for gesture detection and recognition.

[0132] The power supply 1516 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1516 can provide power to the computing device 1500 to enable the components of the computing device 1500 to operate.

[0133] The presentation component 1518 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1518 can receive data from other components (e.g., the GPU 1508, the CPU 1506, etc.) and output the data (e.g., as images, video, sound, etc.).

[0134] Example data center

[0135] Figure 16 An example data center 1600 that can be used in at least one embodiment of the present disclosure is shown. The data center 1600 can include a data center infrastructure layer 1610, a framework layer 1620, a software layer 1630, and / or an application layer 1640.

[0136] As Figure 16 shown, the data center infrastructure layer 1610 can include a resource orchestrator 1612, grouped computing resources 1614, and node computing resources (“node C.R.s”) 1616(1)-1616(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1616(1)-1616(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”) (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and / or cooling modules, etc. In some embodiments, one or more of the node C.R.s 1616(1)-1616(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1616(1)-1616(N) can include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the node C.R.s 1616(1)-1616(N) can correspond to a virtual machine (VM).

[0137] In at least one embodiment, the grouped computing resources 1614 may include individual groups of node CRs 1616 housed within one or more racks (not shown), or multiple racks housed within a data center at different geographical locations (also not shown). Individual groups of node CRs 1616 within the grouped computing resources 1614 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 1616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0138] Resource coordinator 1612 may be configured or otherwise control one or more nodes CRs 1616(1)-1616(N) and / or grouped computing resources 1614. In at least one embodiment, resource coordinator 1612 may include a Software Design Infrastructure (“SDI”) management entity for data center 1600. Resource coordinator 1612 may include hardware, software, or some combination thereof.

[0139] In at least one embodiment, such as Figure 16 As shown, framework layer 1620 may include job scheduler 1628, configuration manager 1634, resource manager 1636, and / or distributed file system 1638. Framework layer 1620 may include a framework for software 1632 supporting software layer 1630 and / or one or more applications 1642 supporting application layer 1640. Software 1632 or application 1642 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1620 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1638 for large-scale data processing (e.g., "big data"). TM(“Spark” hereinafter). In at least one embodiment, the job scheduler 1628 can include a Spark driver to facilitate scheduling workloads supported by different tiers of the data center 1600. The configuration manager 1634 can be capable of configuring different tiers, such as the software tier 1630 and the framework tier 1620 including Spark and a distributed file system 1638 for supporting large scale data processing. The resource manager 1636 can be capable of managing clustered or grouped computing resources mapped to or allocated for supporting the distributed file system 1638 and the job scheduler 1628. In at least one embodiment, the clustered or grouped computing resources can include the grouped computing resources 1614 of the data center infrastructure layer 1610. The resource manager 1636 can coordinate with the resource orchestrator 1612 to manage these mapped or allocated computing resources.

[0140] In at least one embodiment, software 1632 included in the software tier 1630 can include software used by at least portions of the node C.R.s 1616(1)-1616(N), the grouped computing resources 1614, and / or the distributed file system 1638 of the framework tier 1620. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0141] In at least one embodiment, applications 1642 included in the application tier 1640 can include one or more types of applications used by at least portions of the node C.R.s 1616(1)-1616(N), the grouped computing resources 1614, and / or the distributed file system 1638 of the framework tier 1620. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0142] In at least one embodiment, any of the configuration manager 1634, the resource manager 1636, and the resource orchestrator 1612 can implement any number and type of self-modifying actions based on any quantity and type of data acquired in any technically feasible manner. Self-modifying actions can free data center operators of the data center 1600 from making possibly poor configuration decisions and possibly avoid underutilization and / or poor performance portions of the data center.

[0143] According to one or more embodiments described herein, data center 1600 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by using the software and / or computing resources described above with respect to data center 1600 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1600 by using weight parameters computed through one or more training techniques, such as, but not limited to, those described herein.

[0144] In at least one embodiment, the data center 1600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0145] Example network environment

[0146] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 15 This is implemented on one or more instances of computing devices 1500—for example, each device may include similar components, features, and / or functions of one or more computing devices 1500. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1600, examples of which are described in this document. Figure 16 To describe in more detail.

[0147] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0148] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0149] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system.

[0150] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these different functionalities can be distributed across multiple locations from central or core servers (e.g., one or more data centers that can be distributed across a state, a region, a country, globally, and the like). The core servers can designate at least a portion of the functionality to edge servers if the connection to the user (e.g., client device) is relatively close to the edge servers. The cloud-based network environment can be private (e.g., limited to a single organization), can be public (e.g., available to many organizations), and / or combinations thereof (e.g., a hybrid cloud environment).

[0151] One or more client devices can include those described herein with respect to Figure 15At least some of the components, features and functionality of the described one or more example computing devices 1500 are described. By way of example and not limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smart phone, tablet computer, smart watch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, spaceship, virtual machine, drone, robot, hand-held communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0152] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0153] As used herein, the term “and / or” with respect to two or more elements should be interpreted as only one element or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. In addition, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0154] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the disclosure. Rather, the inventive subject matter contemplates that the claimed subject matter can also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0155] Example Paragraph

[0156] A: A method comprising: generating, based at least on audio data representing an utterance, text data corresponding to the utterance; generating, using one or more first models and based at least on the text data, output data indicating whether each token corresponding to the text data is associated with an end of a sentence within the utterance or an end of the utterance; determining, based at least on the output data, a first location in the text data associated with the end of the sentence and a second location in the text data associated with the end of the utterance; and processing, using one or more second models and based at least on the first location and the second location, a first portion of the text data corresponding to the sentence of the utterance and then a second portion of the text data corresponding to a remaining portion of the utterance.

[0157] B: The method of paragraph A, wherein the output data represents at least: a first probability indicating whether each token is associated with the end of the sentence; a second probability indicating whether each token is associated with the end of the utterance; and a third probability indicating whether each token is not associated with the end of the sentence or the end of the utterance.

[0158] C: The method of paragraph A or paragraph B, further comprising: generating, using one or more second models and based at least on the text data, second output data representing whether each token is associated with a lowercase word or an uppercase word, wherein determining the first location and the second location is further based at least on the second output data.

[0159] D: The method of any of paragraphs A-C, further comprising: generating, using the one or more second models and based at least on the text data, second output data representing whether each token is associated with one or more types of punctuation, wherein determining the first location and the second location is further based at least on the second output data.

[0160] E: The method of any of paragraphs A-D, further comprising: generating one or more first embeddings using one or more first encoders and based on at least the audio data; generating one or more second embeddings using one or more second encoders and based on at least the textual data; and generating input data based on at least the one or more first embeddings and the one or more second embeddings, wherein generating the output data uses the one or more models and is based on at least the input data.

[0161] F: The method of any of paragraphs A-E, wherein the first portion of the textual data is processed using the one or more second models based on at least determining the first location and prior to determining the second location.

[0162] G: A system comprising: one or more processors to: determine, using one or more models and based on at least textual data associated with one or more words, an output that indicates whether the one or more words are associated with an end of a sentence and whether the one or more words are associated with an end of an utterance; and cause processing of at least a portion of the textual data based on at least the output.

[0163] H: The system of paragraph G, wherein the one or more processors are further to: determine, based on at least the output, that a first word of the one or more words is associated with the end of the sentence; determine the at least a portion of the textual data based on at least the first word being associated with the end of the sentence; determine, based on at least the output, that a second word of the one or more words is associated with the end of the utterance; determine at least a second portion of the textual data based on at least the second word being associated with the end of the utterance; and cause processing of the at least the second portion of the textual data.

[0164] I: The system of paragraph H, wherein the at least a portion of the textual data is processed prior to the at least the second portion of the textual data.

[0165] J: The system of any of paragraphs G-I, wherein the output indicates at least: one or more first probabilities that indicate whether the one or more words are associated with the end of the sentence; and one or more second probabilities that indicate whether the one or more words are associated with the end of the utterance.

[0166] K: The system of any of paragraphs G-J, wherein the one or more words comprise a plurality of words, and wherein the output indicates at least: one or more first indicators that indicate one or more first words of the plurality of words are associated with an end of the sentence; and one or more second indicators that indicate one or more second words of the plurality of words are associated with an end of the utterance.

[0167] L: The system of any of paragraphs G-K, wherein the one or more processors are further to: determine, using one or more second models and based at least on the textual data, a second output that indicates whether the one or more words are at least one of lower case or upper case, wherein the processing of the at least the portion of the textual data is further based at least on the second output.

[0168] M: The system of paragraph L, wherein the second output indicates at least: one or more first probabilities that indicate whether the one or more words are lower case; and one or more second probabilities that indicate whether the one or more words are upper case.

[0169] N: The system of any of paragraphs G-M, wherein the one or more processors are further to: determine, using one or more second models and based at least on the textual data, a second output that indicates whether the one or more words are associated with one or more types of punctuation, wherein the processing of the at least the portion of the textual data is further based at least on the second output.

[0170] O: The system of any of paragraph N, wherein the second output indicates at least: one or more first probabilities that indicate whether the one or more words are associated with one or more first types of punctuation; and one or more second probabilities that indicate whether the one or more words are associated with one or more second types of punctuation.

[0171] P: The system of any of paragraphs G-O, wherein the one or more processors are further to: generate, using one or more encoders and based at least on audio data, one or more first embeddings; generate, using the one or more encoders and based at least on the textual data, one or more second embeddings; and generate, based at least on the one or more first embeddings and the one or more second embeddings, input data, wherein the determination of the output is based at least on the input data.

[0172] Q: The system of any of paragraphs G-P, wherein: the textual data represents one or more tokens associated with the one or more words; and the output indicates whether the one or more tokens are associated with an end of the sentence or whether the one or more tokens are associated with an end of the utterance.

[0173] R: The system of any of paragraphs G-Q, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulations; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0174] S: One or more processors comprising: processing circuitry to process at least a first portion of textual data at a first instance and a second portion of the textual data at a second instance based at least on an output indicating that the first portion of the textual data is associated with an end of a sentence and the second portion of the textual data is associated with an end of an utterance containing the sentence.

[0175] T: The one or more processors of paragraph S, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Claims

1. A method comprising: generating, based at least on audio data representing an utterance, textual data corresponding to the utterance; generating, using one or more first models and based at least on the textual data, output data indicating whether each token corresponding to the textual data is associated with an end of a sentence within the utterance or an end of the utterance; determining, based at least on the output data, a first location in the textual data associated with the end of the sentence and a second location in the textual data associated with the end of the utterance; and processing, using one or more second models and based at least on the first location and the second location, a first portion of the textual data corresponding to the sentence of the utterance and then a second portion of the textual data corresponding to a remaining portion of the utterance.

2. The method of claim 1, wherein the output data represents at least: a first probability indicating whether each token is associated with the end of the sentence; a second probability indicating whether each token is associated with the end of the utterance; and a third probability indicating whether each token is not associated with the end of the sentence and the end of the utterance.

3. The method of claim 1, further comprising: generating, using one or more second models and based at least on the textual data, second output data representing whether each token is associated with a lowercase word or an uppercase word, wherein determining the first location and the second location is further based at least on the second output data.

4. The method of claim 1, further comprising: generating, using the one or more second models and based at least on the textual data, second output data representing whether each token is associated with one or more types of punctuation, wherein determining the first location and the second location is further based at least on the second output data.

5. The method of claim 1, further comprising: generating, using one or more first encoders and based at least on the audio data, one or more first embeddings; generating, using one or more second encoders and based at least on the textual data, one or more second embeddings; and generating, based at least on the one or more first embeddings and the one or more second embeddings, input data, wherein generating the output data uses the one or more models and is based at least on the input data.

6. The method of claim 1, wherein the first portion of the textual data is processed using the one or more second models based at least on determining the first location and prior to determining the second location.

7. A system comprising: one or more processors to: determine, using one or more models and based at least on textual data associated with one or more words, an output indicating whether the one or more words are associated with an end of a sentence and whether the one or more words are associated with an end of an utterance; and cause at least a portion of the textual data to be processed based at least on the output. ​ ​ ​ 8. The system of claim 7, wherein the one or more processors are further to: determine, based at least on the output, that a first word of the one or more words is associated with an end of the sentence; determine the at least the portion of the textual data based at least on the first word being associated with the end of the sentence; determine, based at least on the output, that a second word of the one or more words is associated with an end of the utterance; determine at least a second portion of the textual data based at least on the second word being associated with the end of the utterance; and cause processing of the at least the second portion of the textual data.

9. The system of claim 8, wherein the at least the portion of the textual data is processed prior to the at least the second portion of the textual data.

10. The system of claim 7, wherein the output indicates at least: one or more first probabilities indicative of whether the one or more words are associated with the end of the sentence; and one or more second probabilities indicative of whether the one or more words are associated with the end of the utterance. the output indicates at least:

11. The system of claim 7, wherein the one or more words comprise a plurality of words, and wherein, one or more first indicators indicative of one or more first words of the plurality of words being associated with the end of the sentence; and one or more second indicators indicative of one or more second words of the plurality of words being associated with the end of the utterance.

12. The system of claim 7, wherein the one or more processors are further to: determine, using one or more second models and based at least on the textual data, a second output indicative of whether the one or more words are at least one of lower case or upper case, the processing of the at least the portion of the textual data is further based at least on the second output.

13. The system of claim 12, wherein the second output indicates at least: wherein one or more first probabilities indicative of whether the one or more words are lower case; and one or more second probabilities indicative of whether the one or more words are upper case.

14. The system of claim 7, wherein the one or more processors are further to: determine, using one or more second models and based at least on the textual data, a second output indicative of whether the one or more words are associated with one or more types of punctuation, the processing of the at least the portion of the textual data is further based at least on the second output.

15. The system of claim 14, wherein the second output indicates at least: wherein one or more first probabilities indicative of whether the one or more words are associated with one or more first types of punctuation; and one or more second probabilities indicative of whether the one or more words are associated with one or more second types of punctuation.

16. The system of claim 7, wherein the one or more processors are further to: ​ ​ generating one or more first embeddings based at least on the audio data using one or more encoders; generating one or more second embeddings based at least on the text data using the one or more encoders; and generating input data based at least on the one or more first embeddings and the one or more second embeddings, wherein the determination of the output is based at least on the input data.

17. The system of claim 7, wherein: the text data represents one or more tokens associated with the one or more words; and the output indicates whether the one or more tokens are associated with an end of the sentence or whether the one or more tokens are associated with an end of the utterance.

18. The system of claim 7, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

19. One or more processors comprising: processing circuitry to process at least a first portion of text data at a first instance and a second portion of the text data at a second instance based at least on an output indicating that the first portion of the text data is associated with an end of a sentence and the second portion of the text data is associated with an end of an utterance containing the sentence, wherein the output is generated based at least on processing the text data using one or more models.

20. The one or more processors of claim 19, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transport simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using edge devices; a system implemented using robots; a system for performing one or more generative AI operations; Systems for performing operations using one or more large language models (LLMs); Systems for performing operations using one or more visual language models (VLMs); Systems for performing one or more conversational AI operations; Systems for generating synthetic data; Systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems including one or more virtual machines (VMs); Systems implemented at least in part in a data center; or Systems implemented at least in part using cloud computing resources.

Images