Generating unified text using speech recognition models for conversational AI systems and applications

Abstract

This paper describes, through various examples, the generation of unified text using speech recognition models for AI systems and applications. Systems and methods are disclosed that utilize a machine learning model trained to generate unified text mapped to user speech, where the unified text includes punctuation, capitalization, inverse text normalization formatting, end-of-sentence (EOS) captures, and / or end-of-utterance (EOU) captures. For example, the machine learning model can receive audio data representing speech as input. The machine learning model can then process the audio data and, based at least on this processing, generate output data mapped to the speech.In some examples, the output data can be tokens, such as tokens associated with automatic speech recognition processing, punctuation and capitalization processing, EOS and / or EOU processing, and / or inverse text normalization processing. In such examples, the tokens can then be processed to generate the unified text.

Description

BACKGROUND

[0001] Automatic speech recognition (ASR) systems are used to process user speech and convert it into text. However, the raw text output by ASR systems is in spoken form, which lacks reliability due to the use of spoken words (e.g., text saying "one hundred dollars" instead of "$100"), missing capitalization, and / or missing punctuation. Therefore, additional systems are often used to further process the raw text, such as a system that inserts punctuation, corrects the text to be in capitalized form, and / or converts the spoken form of words into written form (e.g., performs inverse text normalization).Additionally, under certain circumstances, such as when the text is used by one or more downstream applications, like natural language understanding applications and / or machine translation applications, the text may be further processed using a system that identifies sentence ends and / or utterance ends associated with the language.

[0002] However, problems can arise when using such an architecture, which includes multiple systems that process the audio data and / or text to produce a final text format. Each system, for example, may be prone to outputting errors, such as ASR systems, which output text containing incorrect words. Furthermore, because these systems operate sequentially, if an initial system in the sequence, such as the ASR system, outputs an error, the error can propagate through the remaining systems, causing degradation in the output. Moreover, because the systems operate sequentially by processing the output from the preceding systems, the overall latency of the architecture increases based on the number of systems used to process the audio data.Each system can, for example, include a specific processing latency, where the overall latency of the architecture can be the sum of the latencies of the systems. SUMMARY

[0003] Embodiments of the present disclosure relate to the generation of unified text using speech recognition models for AI systems and applications. Systems and methods are disclosed that utilize a machine learning model trained to generate unified text mapped to user speech, wherein the unified text includes punctuation, capitalization of words, inverse text normalization (ITN) formatting, sentence-end detection (EOS detection), and / or utterance-end detection (EOU detection). The machine learning model can, for example, receive audio data representing user speech as input. In some examples, the audio data is initially processed before being fed into the machine learning model to generate input data representing embeddings that are mapped to frames of the audio data.The machine learning model can then process the audio data and / or the input data and, based at least on this processing, generate output data associated with the unified text. In some examples, the output data might be tokens, such as tokens associated with automatic speech recognition (ASR) processing, punctuation and capitalization (PAC) processing, EOS processing, EOU processing, and / or ITN processing. In such examples, the tokens can then be processed to generate the unified text.

[0004] Unlike conventional systems, the systems of this disclosure can use a single machine learning model that generates the output data representing the unified text, which includes punctuation, capitalization of words, ITN formatting, EOS captures, and / or EOU captures. The systems of this disclosure can therefore reduce the number of systems, models, modules, applications, and / or other processing components required to generate such text in a final format usable by other applications and / or users.As described herein, by reducing the number of processing components, the systems of the present disclosure can also reduce the number of errors associated with processing, since it is only a single model that tends to output errors instead of multiple processing components, and / or reduce the overall latency associated with processing.

[0005] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not fall within the scope of the claims may be described herein.

[0006] Several examples are disclosed concerning the generation of unified text using speech recognition models for AI systems and applications. Systems and methods are disclosed that utilize a machine learning model trained to generate unified text mapped to user speech, where the unified text includes punctuation, capitalization, inverse text normalization formatting, sentence-end captures, and / or utterance-end captures (EOU captures). The machine learning model can, for example, receive audio data representing speech as input. The machine learning model can then process the audio data and, based at least on this processing, generate output data mapped to the speech.In some examples, the output data may be tokens, such as tokens associated with automatic speech recognition processing, punctuation and capitalization processing, EOS and / or EOU processing, and / or inverse text normalization processing. In such examples, the tokens can then be processed to generate the unified text.

[0007] The revelation extends to every new aspect or characteristic described and / or illustrated herein.

[0008] Further features of the disclosure are characterized by the independent and dependent claims.

[0009] Any feature of one aspect of the disclosure can be applied to other aspects of the disclosure in any suitable combination. In particular, procedural aspects can be applied to device or system aspects and vice versa.

[0010] Furthermore, features implemented in hardware can be implemented in software and vice versa. Any reference to software and hardware features herein should be interpreted accordingly.

[0011] Each system or device feature as described herein can also be provided as a process feature, and vice versa. System and / or device aspects described functionally (including means plus functional features) can alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and allocated working memory.

[0012] It is also understood that special combinations of the various features described and defined in aspects of the revelation can be implemented and / or supplied and / or used independently.

[0013] The disclosure also provides computer programs and computer program products comprising software code which, when executed on a data processing device, is adapted to perform one of the procedures and / or to embody one of the device and system features described herein, including all component steps of any procedure.

[0014] The disclosure also provides a computer or computing system (including networked or distributed systems) comprising an operating system that provides a computer program for performing the procedures described herein and / or for embodying any device or system features described herein.

[0015] The revelation also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0016] The revelation also provides a signal that carries one or more of the aforementioned computer programs.

[0017] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0018] Aspects and embodiments of the present disclosure will now be described purely by way of example with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present systems and methods for generating unified text using speech recognition models for AI systems and applications are described in detail below with reference to the accompanying drawings, wherein: Fig. 1A illustrates an example of a process for generating unified text using a machine learning model according to some embodiments of the present disclosure: Fig. 1B illustrates an example of a machine learning model that, according to some embodiments of the present disclosure, can be trained to generate unified text; Fig. 2 an example of generating sets of tokens for frames of audio data according to some embodiments of the present disclosure is illustrated; Fig. 3 illustrates an example of techniques that, according to some embodiments of the present disclosure, can be used to select tokens associated with a frame and / or a group of frames; The Fig. 4A-4B Illustrate examples of different types of processing according to some embodiments of the present disclosure, which are performed by a machine learning model; Fig. 5A illustrates a data flow diagram that demonstrates a process for training a machine learning model according to different embodiments of the present embodiment to perform different types of processing; Fig. 5B illustrates an example of generating ground-truth data for training a machine learning model according to some embodiments of the present disclosure; Fig. 6 An example of a system is illustrated which can perform one or more of the processes described herein according to some embodiments of the present disclosure; Fig. 7 a data flow diagram illustrates a method for processing audio data using a machine learning model that performs ASR processing and ITN processing according to some embodiments of the present disclosure; Fig. 8 a data flow diagram illustrating a method for processing audio data using a machine learning model that performs ASR processing and EOS and / or EOU processing according to some embodiments of the present disclosure; Fig. 9A is a block diagram of an exemplary generative language model system suitable for use in implementing some embodiments of the present disclosure; Fig. 9B is a block diagram of an exemplary generative language model that includes a transformer-encoder-decoder suitable for use in implementing some embodiments of the present disclosure; Fig. 9C is a block diagram of an exemplary generative language model that includes a decoder-transformer-only architecture suitable for use in implementing some embodiments of the present disclosure. Fig. 10 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 11 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] This document discloses systems and methods relating to the generation of unified text using speech recognition models for AI systems and applications. For example, a system can generate, acquire, receive, determine, and / or retrieve audio data representing speech from a user. As described herein, the speech can be associated with an utterance, such as an utterance that includes "We'll rent you a GPU. That'll be one hundred dollars" (and / or any other utterance). In some examples, the audio data can then be preprocessed using one or more processing components, which can be referred to as an "audio processor." The audio processor can be configured, for example, to process the audio data sequentially to separate the audio data into audio frames.The audio processor(s) can then process the audio frames sequentially to generate features such as Mel frequency spectrum features, Fourier transform features, and / or any other type of audio feature. Additionally, the multiple audio processor(s) can further generate embeddings and / or vectors that then represent the features of the audio frame.

[0021] In examples where the audio data is preprocessed using the audio processor(s), input data representing the features, embeddings, and / or vectors can be fed into a machine learning model. In other examples, however, the audio data can be fed directly into the machine learning model, which then processes the audio data using one or more of the processes described herein with respect to the audio processor(s). For example, the machine learning model may include one or more encoders configured to process the audio data and generate the embeddings and / or vectors associated with the audio frames.In any of these examples, the machine learning model can be trained to perform different types of processing, such as ASR processing, PAC processing, EOS processing, EOU processing and / or ITN processing, to produce unified text that includes punctuation, capitalization, EOS information, EOU information and / or ITN formatting.

[0022] Based on at least the processing of input data and / or audio data, the machine learning model can, for example, be trained to generate output data representing tokens associated with speech. In some examples, the tokens can be assigned to the different types of processing for which the machine learning model is trained. The tokens can include, for example, ASR tokens associated with ASR processing, representing at least letters, word segments, and / or words; and PAC tokens associated with PAC processing, representing punctuation marks (e.g., periods, commas, question marks, exclamation marks, etc.).) and / or, tokens (EOS tokens) assigned to EOS processing, representing symbols corresponding to sentence ends, tokens (EOU tokens) assigned to EOU processing, representing symbols corresponding to utterance ends, and / or tokens (ITN tokens) assigned to ITN processing, representing numbers, symbols associated with words (e.g., $ for dollar, ° for degree, etc.), and / or any other written word characters.

[0023] In some examples, the machine learning model can be trained to generate sets of tokens for each frame. For example, the machine learning model can generate a first set of tokens for a first frame, a second set of tokens for a second frame, a third set of tokens for a third frame, and so on. In other examples, the machine learning model can be trained to generate sets of tokens for groups of frames (e.g., two frames, five frames, ten frames, etc.). For example, the machine learning model can generate a first set of tokens for a first group of frames, a second set of tokens for a second group of frames, a third set of tokens for a third group of frames, and so on. In any of the examples, a set of tokens can include one or more tokens (e.g., each token) for which the machine learning model is trained to predict.A set of tokens can include ASR tokens, PAC tokens, EOS tokens, EOU tokens and / or ITN tokens.

[0024] In some examples, the machine learning model can further be trained to generate output data representing probabilities associated with the tokens. For example, for a set of tokens, the machine learning model can output a respective probability indicating the likelihood that a given token is associated with a frame and / or a group of frames. In some examples, the probabilities can be assigned to a range, such as 0 to 1, 0 to 100, and / or any other range of values.

[0025] In some examples, one or more processing components (and / or one or more decoders of the machine learning model) may be configured to use the output data to generate unified text associated with language, which can be referred to as a "token processor(s)." As described herein, unified text may include text containing punctuation, capitalization, EOS symbols, EOU symbols, and / or ITN formatting (e.g., a normalized format also referred to as "normalized text"). To generate the unified text, the token processor(s) may be configured to use the probabilities of selecting one or more tokens from each set of tokens and then use the selected tokens to generate the unified text. In some examples, the processor(s) may use an initial technique (e.g.,Greedy decoding) is used, which involves selecting a specific token that corresponds to the highest probability from each set of tokens. The token processor(s) can then generate the unified text using the selected tokens. The unified text can include, for example, letters, parts of words, words, punctuation marks, capital letters, EOS symbols, EOU symbols, numbers, word symbols, and / or the like, corresponding to the selected tokens.

[0026] Additionally or alternatively, in some examples, the token processor(s) can use a second technique (e.g., beam search / flash decoding) to select a group of tokens associated with a number of highest probabilities from each set of tokens. In some examples, the token processor(s) can then use the groups of tokens selected from the sets of tokens to determine the unified text. For example, the token processor(s) can then process the groups of tokens together with the letters, parts of words, words, punctuation, capitalization, EOS symbols, EOU symbols, numbers, and / or word symbols to which the tokens are associated, using one or more language models trained to output the unified text.While these examples describe the token processor(s) as processing the output data to generate the unified text, the machine learning model in other examples may furthermore use similar techniques to process the output data to generate the unified text (e.g., using one or more decoders).

[0027] While the examples above describe the machine learning model trained to perform ASR processing, PAC processing, EOU processing, EOS processing, and / or ITN processing, in other examples the machine learning model can be trained to perform one or more of these processes. If the machine learning model is trained as a first example to perform ASR processing, PAC processing, EOU processing, and EOS processing, the tokens issued by the machine learning model can include ASR tokens, EOU tokens, and EOS tokens, and the unified text can include EOU symbols and EOS symbols.If the machine learning model is trained as a second example to perform ASR processing, PAC processing, and ITN processing, the tokens output by the machine learning model can include ASR tokens, PAC tokens, and ITN tokens, and the unified text can include punctuation, capitalization, and / or be in a normalized format.

[0028] In some examples, the system(s) may use one or more techniques to train the machine learning model to generate the output data associated with the unified text. For example, the machine learning model may be trained to output the ASR tokens, PAC tokens, EOS tokens, EOU tokens, and / or ITN tokens. To train the machine learning model, the system may generate, acquire, receive, determine, and / or retrieve training input data representing audio data such as instances of speech (e.g., utterances). Additionally, the system may generate, acquire, receive, determine, and / or retrieve ground truth data associated with the training input data. In some examples, the ground truth data may represent instances of unified text corresponding to the utterances.Additionally or alternatively, in some examples, the ground truth data can represent the instances of unified text in tokenized form, such as including the tokens on which the machine learning model is trained to make predictions.

[0029] The system(s) can then use various techniques to train the machine learning model using training data (e.g., the training input data and corresponding ground truth data). As a first example, if the ground truth data represents the tokens, the machine learning model can process the training input data and, based at least on this processing, generate output data representing tokens. One or more training engines can then update one or more losses based at least on comparing the output tokens with the ground truth tokens and updating one or more parameters and / or weights of the machine learning model using the loss(s).As a second example, if the ground truth data represents instances of unified text, the machine learning model can process the training input data and, based at least on this processing, generate output data representing tokens. The training engine(s) can then update one or more losses based at least on comparing the output unified texts with the ground truth unified text and updating one or more parameters of the machine learning model using the loss(s). While these are just a few exemplary techniques for how the machine learning model can be trained, in other examples the system(s) can train the machine learning model using additional and / or alternative techniques described herein.

[0030] As described herein, the system(s) may use one or more techniques to generate the training data. In some examples, such as generating sufficient training data to initially train the machine learning model, the system(s) may use one or more systems, models, modules, and / or the like to automatically generate the ground truth data. The system(s) may use an ASR system, a PAC system, an EOU / EOS system, and / or an ITN system to generate unified text that includes punctuation, capitalization, EOU symbols, EOS symbols, and / or is in a normalized format. Additionally or alternatively, in some examples, such as generating training data to test the accuracy of the machine learning model, the one or more systems may generate the ground truth data using user feedback.

[0031] In some examples, the machine learning model can include any type of model, neural network (e.g., a convolutional neural network, a recurrent neural network, etc.), transformer, module, and / or processing component configured to perform one or more of the processes described herein. For example, the machine learning model can include any type of model that incorporates an encoder configured to initially process the audio data to generate audio embeds, and a decoder configured to process the audio embeds to generate the output data associated with the unified text. In such an example, the machine learning model can include a feedback loop between the decoder and the encoder to ensure that linguistic cues from partial text can be returned to the encoder.

[0032] In some examples, different types of technologies can use the machine learning model to perform at least some of the processes described herein. As a first example, systems running additional applications, such as natural language applications configured to interpret text (e.g., for one or more systems like a vehicle control system, a robotics control system, an avatar communication system, etc.), machine translation applications configured to translate text from one language to another, and / or language modeling applications configured to process text, can use the machine learning model to generate the unified text that is input into these applications.Since, in such an example, the unified text can specify the positions of sentence ends and / or utterance ends that are associated with language, the systems can input the unified text into the applications using techniques that help improve processing performance. For example, the systems can input sections of the unified text into the applications based on the positions of the sentence ends and / or the positions of the utterance ends.

[0033] As a second example, systems that provide interactive applications, such as gaming applications, communication applications, and / or collaborative group applications, can use the machine learning model to perform one or more processes. For instance, the systems can receive audio data representing speech from a user of an interactive application. The systems can then process the audio using the machine learning model, such as by using one or more of the processes described herein, to generate unified text associated with the speech. Additionally, the systems can then deliver the unified text to one or more devices of one or more users of the interactive application in such a way that the one or more devices are able to both output the audio associated with the speech and display content associated with the unified text.

[0034] In some examples, the machine learning model can be packaged as a microservice, such as an inference microservice (e.g., NVIDIA NIMs), which can include a container (e.g., an operating system-level virtualization package) that can contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. The inference microservice might, for example, include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model can be enclosed within the container itself.Therefore, in some implementations, the machine learning models described herein can be used as an inference microservice to accelerate the deployment of models on any cloud, in any data center, or on an edge computing system, while ensuring data security. The inference microservice may, for example, include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment and execution software, such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime environment and model optimizations that deliver low latency and high throughput for production applications (such as NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g.,(including identity, metrics, health checks, and / or monitoring). The machine learning model(s) described herein may be included as part of the microservice along with an accelerated infrastructure capable of single-command deployment and / or orchestration and automatic scaling using a container orchestration system on accelerated infrastructure (e.g., from a single device to a data center size). Therefore, the inference microservice may include the machine learning model(s) (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity, and other monitoring.In some embodiments, the inference microservice may include software for performing in-place replacement and / or updates of the machine learning model(s). During replacement or update, the software performing the replacement / update may retain user configurations of the inference runtime software and enterprise management software.

[0035] The systems and methods described herein can be used, for example, without restriction, by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, steered and unsteered robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled with one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones and / or other types of vehicles.Furthermore, the systems and methods described herein can be used for a variety of purposes, including but not limited to machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actuator simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0036] The disclosed embodiments can be used in a variety of different systems, such as automotive systems (e.g.a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine, systems implemented using a robot, aeronautical systems, media systems, boating systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems implementing large language models (LLMs), systems implementing one or more vision language systems (VLMs), systems implementing one or more multimodal models, systems using or employing one or more inference microservices, systems that integrate one or more machine learning models in a service or microservice together with an OS-level virtualization package (e.g.,a container), systems containing one or more virtual machines (VMs), systems implementing one or more multimodal models, systems for performing operations to generate synthetic data, systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems for performing generative AI operations, systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0037] With reference to Fig. 1A illustrates Fig. Figure 1A shows an example of a process 100 for generating unified text using a machine learning model according to some embodiments of the present disclosure. It is understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional units that may be implemented as separate or distributed components, or in conjunction with other components, in any suitable combination and at any suitable location. Various functions described herein as being performed by units may be performed by hardware, firmware, and / or software.For example, various functions can be performed by a processor that executes instructions stored in the main memory.

[0038] Process 100 can include one or more audio processors 102 that receive audio data 104 representing speech. As described herein, the speech can be associated with an utterance, such as an utterance that includes "We'll rent you a GPU. That'll be one hundred dollars." Process 100 can then include the audio processor(s) 102 processing the audio data 104 to generate input data 106 for a machine learning model 108. The audio processor(s) 102 can, for example, be configured to process the audio data 104 sequentially to separate the audio data into audio frames. As described herein, an audio frame can contain any length, such as 25 milliseconds, 50 milliseconds, 75 milliseconds, and / or any other length segment of the audio data 104.The audio processor(s) 102 can then process the audio frames sequentially to generate features such as Mel frequency spectrum features, Fourier transform features, and / or any other type of audio features representing speech. Additionally, the audio processor(s) 102 can generate embeddings and / or vectors that then represent the features of the audio frames, where the input data 106 can represent the audio frames, the features, the embeddings, and / or the vectors.

[0039] As described herein, the audio processor(s) 102 may include any type of processing component configured to perform one or more of the processes described herein, such as one or more machine learning models, one or more neural networks, one or more encoders, one or more modules, one or more fast Fourier transform components, and / or the like. While the example of the Fig. 1A illustrates the audio processor(s) 102 as separate from the machine learning model 108; however, the machine learning model 108 can additionally include the audio processor(s) 102 in other examples. For example, and as shown in the example of Fig. As illustrated in Figure 1B, the machine learning model 108 can include one or more encoders configured to perform at least part of the processing described herein with reference to the audio processor(s) 102.

[0040] The process 100 can then include the machine learning model 108 processing the input data 106 (and / or, in some examples, the audio data 104) and, based at least on the processing, generating output data 110 that is associated with the language. In some examples, and as from the example of the Fig. As illustrated in Figure 1A, the output data 110 can represent at least tokens 112(1)-(4) (hereinafter also referred to in the singular as "Token 112" or in the plural as "Tokens 112") and probabilities 114(1)-(4) (also referred to in the singular as "Probability 114" or in the plural as "Probabilities 114"). In some examples, the tokens can be assigned to the different types of processing for which the machine learning model 108 is trained. The tokens 112 may, for example, include at least ASR tokens 112(1) that are allocated to ASR processing, PAC tokens 112(2) that are allocated to PAC processing, EOS / EOU tokens 112(3) that are allocated to EOS / EOU processing and / or ITN tokens 112(4) that are allocated to ITN processing.

[0041] In some examples, the tokens 112, which are associated with the different types of processing, can represent unique text types such that the machine learning model 108 is able to generate the unified text described herein. For example, the ASR tokens 112(1) can represent at least letters, characters, sentences (or sections thereof), sections of words (partial words), and / or words such that the machine learning model 108 can perform ASR processing. Additionally, the PAC tokens 112(2) can represent at least punctuation marks (e.g., periods, commas, question marks, exclamation marks, etc.) and / or uppercase letters such that the machine learning model 108 can perform PAC processing.Furthermore, the EOS / EOU tokens 112(3) can represent at least one or more symbols associated with sentence ends and / or one or more symbols associated with utterance ends, so that the machine learning model 108 can perform EOS / EOU processing. In addition, the ITN tokens 112(4) can represent at least numbers, symbols associated with words (e.g., $ for dollars, ° for degrees, etc.), and / or any other written word characters, so that the machine learning model 108 can perform ITN processing.

[0042] In some examples, the machine learning model 108 can be trained to generate sets of tokens for each frame. For example, the machine learning model 108 can generate a first set of tokens 112 for a first frame, a second set of tokens 112 for a second frame, a third set of tokens 112 for a third frame, and so on. In some examples, the machine learning model 108 can be trained to generate sets of tokens 112 for groups of frames (e.g., two frames, five frames, ten frames, etc.). For example, the machine learning model 108 can generate a first set of tokens 112 for a first group of frames, a second set of tokens 112 for a second group of frames, a third set of tokens 112 for a third group of frames, and so on. In any of the examples, a set of tokens can contain one or more tokens (e.g.,Each token 112) for which the machine learning model is trained to make predictions. A set of tokens 112 can, for example, include the ASR tokens 112(1), the PAC tokens 112(2), the EOS / EOU tokens 112(3) and / or the ITN tokens 112(4).

[0043] Fig. Figure 2 illustrates, for example, an example of generating sets of tokens for frames of audio data according to some embodiments of the present disclosure. As shown, the machine learning model 108 can generate output data (e.g., the output data 110) that includes at least first tokens 202(1) associated with one or more of the first frames 204(1) of audio data (e.g., audio data 104), second tokens 202(2) associated with one or more of the second frames 204(2) of the audio data, third tokens 202(3) associated with one or more of the third frames 204(3) of the audio data, and so on, up to final tokens 202(N) associated with one or more of the final frames 204(N) of the audio data. As described herein, in some examples, one or more sets (e.g.,each set) of the tokens 202(1)-(N) include the ASR tokens 112(1), the PAC tokens 112(2), the EOS / EOU tokens 112(3) and / or the ITN tokens 112(N).

[0044] With renewed reference to Fig. 1A can specify the probabilities that the tokens are assigned to the frame and / or groups of frames. In some examples, the machine learning model can be trained to generate a probability assigned to each of the tokens. In other examples, however, the machine learning model can be trained to generate a probability for a group of tokens. Additionally, in some examples, the probabilities can be assigned to a range, such as 0 to 1, 0 to 100, and / or any other range of values.

[0045] How to proceed from the example of Fig. As illustrated in Figure 1A, the process 100 can include one or more token processors 116 that process the output data 110 and, based at least on the processing, generate text data 118 representing the unified text associated with the language. The token processor(s) can, for example, be configured to use the probabilities 114 for selecting one or more tokens 112 from each set of tokens 112 and then use the selected token(s) 112 to generate the unified text. In some examples, the token processor(s) can use an initial technique (e.g., greedy decoding) to select the tokens 112, which involves selecting a particular token 112 associated with the highest probability from each set of tokens 112. Additionally or alternatively, the token processor(s) 116 may use a second technique for selecting the tokens in some examples (e.g.B. Beam-Search / Flash-Decoding), which includes selecting a group of tokens that is assigned to a number of highest probabilities 114 from each set of tokens.

[0046] Fig. Figure 3 illustrates, for example, an instance of techniques that, according to some embodiments of the present disclosure, can be used to select tokens to be associated with a frame and / or a group of frames. As shown, the tokens 302(1)-(O) (hereafter referred to in the singular as "Token 302" or in the plural as "Tokens 302") can be assigned probabilities 304(1)-(O) (also referred to in the singular as "Probability 304" or in the plural as "Probabilities 304"). In the example of the Fig. 3. The tokens 302 can be ordered from the highest probability 304, which is assigned to the first token 302(1), and in descending order of probabilities 304 to the final token 302(0). A first technique 306, such as a greedy decoding technique, can therefore include selecting the first token 302(1), which is assigned to the highest probability 304(1). Additionally, a second technique 308, such as a beam search / flash decoding technique, can include selecting the group of tokens 302(1)-(5), which is assigned to the five highest probabilities 304(1)-(5). While the example of the Fig. 3. As illustrated by selecting the group as including five tokens 302(1)-(5) that are assigned to the 5 highest probabilities 304(1)-(5), in other examples the group can include any number of tokens 302 that are assigned to any number of highest probabilities 304.

[0047] Referring again to the example of the Fig. 1A, the token processor(s) 316 can then use the selected tokens 112 to generate the unified text. As a first example and for the first technique for selecting the tokens 112, the token processor(s) 116 can generate the unified text such that it includes the letters, parts of words, words, punctuation marks, capital letters, EOS symbols, EOU symbols, numbers, symbols assigned to words, and / or the like that are assigned to the selected tokens 112. As a second example and for the second technique, the token processor(s) 116 can process the groups of tokens 112 together with the letters, parts of words, words, punctuation marks, capital letters, EOS symbols, EOU symbols, numbers, symbols associated with words and / or the like associated with the tokens 112, using one or more language models 120.Based at least on the processing, the language model(s) 120 can be configured to select a sequence of tokens 112 that generates the most meaningful unified text (e.g., including correct language).

[0048] As described herein, by carrying out the process 100 of the Fig. 1A The Unified Text represented by the text data 118 can be assigned ASR processing, PAC processing, EOU / EOS processing, and / or ITN processing. For example, if the machine learning model 108 is trained as a first example to perform ASR processing and EOU / EOS processing, the tokens issued by the machine learning model 108 can include ASR tokens 112(1) and EOU / EOS tokens 112(3), and the Unified Text can include EOU symbols and EOS symbols. When the machine learning model 108 is trained as a second example to perform ASR processing, PAC processing and ITN processing, the tokens issued by the machine learning model 108 may include ASR tokens 112(1), PAC tokens 112(2) and ITN tokens 112(4), and the unified text may include punctuation, capitalization and be in a normalized format.

[0049] The Fig. Figures 4A-4B illustrate, for example, different types of processing that can be performed by a machine learning model 108 according to some embodiments of the present disclosure. As shown, in each of the examples of Fig. 4A-4B The machine learning model 108 processes audio data 402, which includes the utterance "We rent you a GPU. That'll be one hundred dollars." Therefore, in the first example, the machine learning model 108 can only be trained to perform ASR processing. The machine learning model 108 can therefore be used to generate text data 404(1) that represents text (referred to as "ASR text") that includes "we rent you a GPU that'll be one hundred dollars." Next, for a second example, the machine learning model 108 can be trained to perform both ASR and ITN processing. The machine learning model 108 can therefore be used to generate initial text data 406(1) that represents initial unified text that includes "we rent you a GPU that'll be $100."In this second example, the first Unified Text includes “$100” instead of “one hundred dollars” from the ASR text in the first example.

[0050] Next, the machine learning model 108 can be trained as a third example to perform both ASR processing and PAC processing. The machine learning model 108 can therefore be used to generate a second text data set 406(2) representing a second unified text that includes "We'll rent you a GPU. That'll be one hundred dollars." In this third example, the second unified text includes correct punctuation and capitalization compared to the ASR text from the first example. Next, the machine learning model 108 can be trained as a fourth example to perform both ASR processing and EOS / EOU processing. The machine learning model 108 can therefore be used to generate a third text data set 406(3) representing a third unified text that includes "we'll rent you a GPU / that'll be one hundred dollars #."In this fourth example, the unified text includes the symbol “ / ” to specify the EOS and the symbol “#” to specify the EOU, in contrast to the ASR text from the first example. However, in other examples, unified text can include any other symbols to specify the EOS and / or the EOU.

[0051] Next, the machine learning model 108 can be trained as a fifth example to perform ASR processing, PAC processing, and ITN processing. The machine learning model 108 can therefore be used to generate fourth text data 406(4), representing the fourth unified text, which includes "We'll rent you a GPU. That'll be $100." In this fifth example, the fourth unified text includes the correct punctuation, correct capitalization, and "$100" compared to the ASR text from the first example. Next, the machine learning model 108 can be trained as a sixth example to perform ASR processing, PAC processing, and EOU / EOS processing. The machine learning model 108 can therefore be used to generate fifth text data 406(5), which represents the fifth unified text, including “We'll rent you a GPU. / That'll be one hundred dollars #.”In this sixth example, the fifth Unified Text includes the correct punctuation, correct capitalization, the symbol “ / ” to indicate the EOS, and the symbol “#” to indicate the EOU, compared to the ASR text from the first example.

[0052] Next, the machine learning model 108 can be trained as a seventh example to perform ASR processing, EOU / EOS processing, and ITN processing. The machine learning model 108 can therefore be used to generate sixth text data 406(6), representing the sixth unified text, which includes "we rent them a GPU / that'll be $100." In this seventh example, the unified text includes the symbol " / " to indicate the EOS and the symbol "#" to indicate the EOU, and "$100" in comparison to the ASR text from the first example. Finally, the machine learning model 108 can be trained as an eighth example to perform ASR processing, PAC processing, EOU / EOS processing, and ITN processing. The machine learning model 108 can therefore be used to generate seventh text data 406(7) representing the seventh unified text, which includes “We'll rent you a GPU. / That'll be $100. #.”In this eighth example, the seventh Unified Text includes the correct punctuation, correct capitalization, the symbol “ / ” to specify the EOS and the symbol “#” to specify the EOU, as well as “$100” compared to the ASR text from the first example.

[0053] Fig. Figure 1B illustrates an example of a machine learning model 122 (which may be similar to and / or represent machine learning model 108) that may be trained to generate unified text according to some embodiments of the present disclosure. As shown, the audio data 104 can be directly input into the machine learning model 122. The machine learning model 122 may then include an encoder 124 configured to process the audio data 104 and generate embeds 126 associated with the frames of the audio data 104, similar to the audio processor(s) 102. Additionally, the machine learning model 122 may include a decoder 128 configured to process the embeds 126 and, based at least on the processing, to generate output data 130 associated with speech.As described herein, decoder 128 may, but is not limited to, include a recurrent decoder, a connectionist temporal classification decoder, a transformer decoder, and / or a decoder of another type. Additionally, output data 130 may represent tokens similar to output data 110, and / or output data 130 may represent unified text similar to text data 118.

[0054] As described herein, the machine learning model 108 can be trained to perform various types of processing, such as ASR processing, PAC processing, EOS / EOU processing and / or ITN processing. Fig. Figure 5A therefore illustrates a data flow diagram that demonstrates a process for training the machine learning model 108 to perform various types of processing according to some embodiments of the present disclosure. As shown, the machine learning model 108 can be trained using training data 502. In some examples, the training data 502 can include instances of speech (e.g., utterances) similar to the audio data 104 and / or vectors and / or embeddings associated with the instances of speech, similar to the input data 106. The training data 502 can be synthetically generated (e.g., produced from computer models or renderings), real-world generated (e.g., conceived and generated from real-world data, such as audio data representing user speech), and / or a combination thereof.

[0055] The machine learning model 108 can be trained using the training data 502 and corresponding ground truth data 504. As shown, in some examples, the ground truth data 504 can include unified text 506, which corresponds to the instances of language associated with the training dataset 502. Additionally or alternatively, in some examples, the ground truth data 504 can include tokens 508 associated with the unified text 506, such as ASR tokens, PAC tokens, EOU / EOS tokens, and / or ITN tokens. As described herein, ground truth data can be synthetically generated (e.g., from computer models or renderings), actually generated (e.g., designed and generated from real data), automatically generated by machines (e.g., using feature analysis and learning to extract features from data and then generate labels), or annotated by humans (e.g.,a labeler or annotation expert) and / or a combination thereof. In some examples, there may be corresponding ground-truth data 504 for each instance of the training data 502 (e.g., each instance of language, such as each utterance).

[0056] In some examples, training data can be obtained, for instance, by using human listeners and linguistics to transcribe audio recordings into unified text, which includes punctuation, capitalization, sentence endings, utterance endings, and inverse text normalization. Additionally or alternatively, in some examples, training data can be obtained using state-of-the-art production model systems, where such a system processes the audio data sequentially to predict text, then punctuation and capitalization, then sentence endings and utterance endings, and then inverse text normalization, or another processing order.In some further examples, a hybrid approach based on a cascade of systems and human-in-the-loop verification for randomly selected training examples can be used to ensure accurate ground truth for training data in order to obtain the training data.

[0057] Fig. Figure 5B illustrates, for example, an example of generating the ground truth data 504 for training a machine learning model 108 according to some embodiments of the present disclosure. As shown, the ground truth data 504 can be generated using a number of processing components, such as an ASR component 510 configured to perform ASR processing, a PAC component 512 configured to perform PAC processing, an EOS / EOU component 514 configured to perform EOS / EOU processing, a language model 516, and / or an ITN component 518 configured to perform ITN processing.

[0058] Audio data 520 (which may represent and / or be similar to the training data 502) may, for example, initially be processed using the ASR component 510 to generate text in spoken form, such as the first text data 404(1) of the example of Fig. 4A. The text can then be processed using the PAC component 512 to insert punctuation into the text and to correct the text so that it is in uppercase, similar to the second text data 406(2) of the example of Fig. 4A. Next, the text can be processed using the EOS / EOU component 514, which is configured to determine the sentence endings and / or utterance endings associated with the text, similar to the fifth text data 406(5) from the example of the Fig. 4B. After capturing the sentence ends and / or utterance ends, the text can be processed using the language model 516 to correct any errors from the ASR component 510 and / or to improve the text's structure. Finally, the text can be processed using the ITN component 518, which is configured to perform ITN processing, such as converting spoken words to written words, similar to the seventh text data 406(7) from the example of Fig. 4B. The output from the processing can then include text data 522, which represents Unified Text.

[0059] While the example of Fig. 5B Processing of the audio data 520 using each of the ASR component 510, the PAC component 512, the EOS / EOU component 514, the speech model 516, and the ITN processing 518 is illustrated. In other examples, the audio data 520 can be processed using one or more of the ASR component 510, the PAC component 512, the EOS / EOU component 514, the speech model 516, and the ITN processing 518. The processing performed on the audio data 520 can, for example, be based on how the machine learning model 108 is trained. As a first example, if the machine learning model 108 is trained only to perform ASR processing and ITN processing, the audio data 520 can be processed using only the ASR component 510 and the ITN component 518, such as to generate unified text similar to the unified text generated from the first text data 106(1) of the example of Fig. 4A is shown. As a second example, if the machine learning model 108 is only trained to perform ASR processing, PAC processing, and ITN processing, the audio data 520 can be processed using only the ASR component 510, the PAC component 512, and the ITN component 518, such as to generate unified text similar to the unified text produced by the fourth text data 106(4). From the example of Fig. 4B is shown.

[0060] Referring again to the example of the Fig. 5A, one or more training engines 524 can use one or more loss functions to train the machine learning model 108. These loss functions measure loss (e.g., errors) in outputs 526 compared to the ground truth data 504. As described herein, in some examples, such as when the ground truth data 504 represents instances of the unified text 506, the outputs 526 may also include instances of unified text as determined using the machine learning model 108, which processes the audio data 520. Therefore, the training engine(s) 524 can compare the instances of the unified text 506 with the instances of text from the outputs 526 to measure the loss(s). Additionally or alternatively, in some examples, such as when the ground truth data 504 represent the tokens 508, the outputs 526 may also include tokens as determined by the machine learning model that processes the audio data 520.Therefore, the training engine(s) 524 can compare the tokens 508 with the tokens from the outputs 526 to measure the loss(s).

[0061] A loss function can be used, for example, for the Unified Ground Truth (Unified Tokens). Specifically, markers (e.g., symbols) associated with PAC, EOS, EOU, and / or ITN can be inserted into the Ground Truth text to obtain the Unified Text 506. Subsequently, Unified Tokens 508 can be generated to represent the Unified Text 506 with the inserted markers, such that the Unified Tokens 508 also represent PAC, EOS, EOU, and / or ITN. Therefore, the loss function can then be calculated using predicted tokens represented by the output data 526 and the Unified Tokens 508 represented by the Ground Truth data 504. Any type of loss function can then be used, such as cross-entropy loss, mean square loss, or any other type of loss.

[0062] In some of these examples, any type of loss function can be used. For instance, the difference function between the unified ground truth text and the predicted unified text can serve as the loss function, depending on the use case and training data. Additionally, in some examples, there is no restriction on the choice of loss function for the proposed approach. Some of the losses that can be used include connectionist temporal classification loss (CTC loss), RNN transducer loss (RNN-T loss), cross-entropy loss, sequence-to-sequence loss, lattice-free maximum mutual information (LF-MMI), minimum word error rate loss (MWER loss), minimum Bayes risk loss (MBR loss), or a weighted combination of these losses.To build our prototype system, a hybrid loss can be used, which is a weighted combination of CTC loss and RNN-T loss, calculated on Unified Ground-Truth text and predictions.

[0063] In some examples, different outputs can have different loss functions. For instance, the ASR tokens 508 and / or the Unified Text 506 associated with ASR processing can use a first loss function, the PAC tokens 508 and / or the Unified Text 506 associated with PAC processing can use a second loss function, the EOS / EOU tokens 508 and / or the Unified Text 506 associated with EOS / EOU processing can use a third loss function, and / or the ITN tokens 508 and / or the Unified Text 506 associated with ITN processing can use a fourth loss function. In such examples, the loss functions can be combined to form an overall loss (where one or more losses can be weighted), and the overall loss can be used to train (e.g., update the parameters) the machine learning model 108.In some examples, backward computations can be performed to recursively calculate the gradients of the loss function(s) with respect to training parameters. In some examples, weights and / or biases of the machine learning model 108 can be used to calculate these gradients.

[0064] As described herein, the machine learning model 108 can be trained to perform one or more specific types of processing. As a first example, if the machine learning model 108 is trained to perform ASR processing and ITN processing, the ground truth data 504 can be specific for ASR processing and ITN processing, such that the unified text 506 can be in ITN format at a minimum and / or the tokens 508 can include ITN tokens. The training engine(s) 524 can therefore determine the loss(s) associated with ITN processing at a minimum, such as by using a word error rate associated with ITN.As a second example, when the machine learning model 108 is trained to perform ASR processing, PAC processing, and ITN processing, the ground truth data can be specific for ASR processing, PAC processing, and ITN processing, such that the unified text can include punctuation, capitalization, and ITN formatting, and / or the tokens 508 can include at least PAC tokens and ITN tokens. The training engine(s) 524 can therefore determine a first loss(s) associated with PAC processing using a first word error rate and a second loss(s) associated with ITN processing using a second word error rate.

[0065] In some examples, different machine learning models can be created and / or trained for different languages, and / or a single machine learning model can be used for multiple languages. For example, single-language models (one ASR model for each language / region) and non-English models (one ASR model for more than one language) can be created. For example, an en-US ASR model is a single-language model focused on US English. An en-GB ASR model is a single-language model focused on British English, and so on. As another example, a multi-language ASR model can be developed for the regions of Europe, the Middle East, and Africa for languages ​​commonly used in these regions, namely English, French, German, Dutch, Italian, Spanish, Portuguese, Arabic, Swahili, and so on.In other words, there may be no limit to the number of models that can support a multilingual model. Multilingual models may require EOS / EOU tag tokens to distinguish themselves from the superset of punctuation symbols used in all the languages ​​served by the multilingual model.

[0066] In some examples, a machine learning model 108 can be created and / or trained to perform code-switching. Code-switching scenarios involve situations in which a user speaks in a first language, then switches to a second language, and then returns to the first language or continues speaking the second language. Under some circumstances, there is no limit to the number of different languages ​​spoken in a code-switching utterance. Additionally, under some circumstances, there is no limit to the frequency with which the language is switched or changed within an utterance. For example, a user can speak English for two minutes, switch to German for the next two minutes, and then switch back to English for one minute.In such an example, a Unified ASR model for code-switching scenarios can output the unified text that corresponds to the spoken language and the text, with the switching of the output language occurring automatically. Additionally, the Unified ASR model can then learn this during training, using code-switching audio and corresponding ground truth text to train / teach the Unified ASR model to recognize unified text from audio.

[0067] In some examples, sets of punctuation symbols can be selected to represent output based on how a developer wants to train machine learning model(s) 108 and / or based on different languages ​​for which machine learning models 108 are being trained. For example, a developer can select specific ways to denote punctuation. A developer might choose to use two spaces for a comma, three spaces for a period, four spaces for a question mark, or a dash for an exclamation mark. By performing one or more of the processes described herein, the machine learning model(s) 108 can be trained to produce text that includes the set of punctuation.The ground truth data 504 used to train the machine learning model(s) 108 may, for example, include a set of punctuation marks for which a developer in Unified Text wants the machine learning model(s) 108 to learn the set of punctuation marks during training.

[0068] Fig. Figure 6 illustrates an example of a system 602 that can perform one or more of the processes described herein according to some embodiments of the present disclosure. As shown, the system 602 (which may represent and / or include a computing device 1000 and / or an exemplary data center 1100) can include one or more processors 604 (which may be similar to and / or include one or more central processing units 1006 and / or one or more graphics processing units 1008), and memory 606 (which may be similar to and / or include memory 1004). The memory 606 can, for example, store the audio processor(s) 102, the machine learning model 108, and the token processor(s) 116.Additionally, the processor(s) can execute the audio processor(s) 102, the machine learning model 108 and / or the token processor(s) 116 to perform one or more of the processes described herein.

[0069] Additionally, as from the example of the Fig. As shown in Figure 6, the system 602 can receive the audio data 104 from one or more client devices 608 (which may also be similar to and / or include an exemplary computing device 1000), and / or send the text data 118 to the client device(s) 608. The client device(s) 608 may, for example, use one or more input devices such as one or more microphones to generate the audio data 104. After generating the audio data 104, the client device(s) 608 may send the audio data 104 to the system 602 for processing.

[0070] Additionally, the client device(s) 608 can perform one or more processes using the text data 118. As a first example, after receiving the text data 118, the client device(s) 608 can use one or more output devices 610 to provide the unified text, such as a display that presents the unified text. As a second example, after receiving the text data 118, the client device(s) 608 can use one or more additional processing components 612 configured to further process the unified text. The processing component(s) 612 can, for example, process the unified text using one or more natural language processing applications, machine translation applications, and / or any other type of processing application. While the example of the Fig. As illustrated in Figure 6, the client device(s) 608 include the processing component(s) 612; in other examples, the system(s) 602 may include the processing component(s) 612.

[0071] In some examples, the machine learning model can be packaged as a microservice, such as an inference microservice (e.g., NVIDIA NIMs), which can include a container (e.g., an operating system-level virtualization package) that can contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. The inference microservice can, for example, include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a sufficiently small number of parameters), the model can be enclosed within the container itself.In some embodiments, the machine learning model described herein can be used as an inference microservice to accelerate the deployment of models on any cloud, in any data center, or on an edge computing system while ensuring data security. The inference microservice may, for example, include one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment and execution software such as NVIDIA's Triton Inference Server), and / or one or more APIs for high-performance deep learning inference, which may include an inference runtime environment and model optimizations that deliver low latency and high throughput for production applications (such as NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g.,(including identity, metrics, health checks, and / or monitoring). The machine learning model 108 described herein can be included as part of the microservice along with an accelerated infrastructure capable of being deployed with a single command and / or orchestrated and automatically scaling with a container orchestration system on an accelerated infrastructure (e.g., from a single device to a data center size). Therefore, the inference microservice can include the machine learning model(s) (e.g., optimized for high-performance inference), inference runtime software for executing the machine learning model 108 and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity, and other monitoring.In some embodiments, the inference microservice may include software to perform in-place replacement and / or updates of the machine learning model 108. During replacement or update, the software performing the replacement / update may retain user configurations of the inference runtime software and enterprise management software.

[0072] With reference to the Fig. 6 and Fig. 7. Each block of procedures 600 and 700 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Procedures 600 and 700 can also be embodied as computer-usable instructions stored in computer storage media. Procedures 600 and 700 can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name only a few. In addition, procedures 600 and 700 are illustrated by reference to Fig. 1A described. However, these procedures 600 and 700 can additionally or alternatively be performed by any system or combination of systems, including, but not limited to, any of the systems described herein.

[0073] Fig. Figure 7 illustrates a data flow diagram that, according to some embodiments of the present disclosure, shows a method 700 for processing audio data using a machine learning model that performs ASR processing and ITN processing. At block B702, the method 700 may include generating embeddings associated with audio data representative of speech. The audio processor(s) 102 may, for example, process the audio data 104 and, based at least on the processing, generate the input data 106 representing the embeddings. As described herein, the embeddings may be associated with frames of the audio data. While the example of the Fig. 1A Illustrating the audio processor(s) 102 as separate from the machine learning model 108, the audio processor(s) 102 may additionally include a section of the machine learning model 108 (e.g., one or more encoders of the machine learning model 108).

[0074] Method 700, at block B704, can generate output data based on at least one machine learning model processing input data associated with the embeddings. This output data can include one or more of the first tokens associated with automatic speech recognition and one or more of the second tokens associated with inverse text normalization. For example, machine learning model 108 can process input data 106 and, based on at least the processing, generate output data 110 representing ASR token(s) 112(1) and ITN token(s) 112(4). As described herein, in some examples, output data 110 can further represent PAC token(s) 112(2) and / or EOS / EOU token(s) 112(3).Additionally, in some examples, the output data 110 may represent one or more probabilities 114(1) associated with the ASR token(s) 112(1), and / or one or more probabilities 114(4) associated with the ITN token(s) 112(4).

[0075] Procedure 700 at block B706 can include generating normalized text associated with the language based on at least one or more first tokens and one or more second tokens. For example, processor(s) 116 can use ASR token(s) 112 and ITN token(s) 112(4) to generate normalized text (e.g., Unified Text) as if it were text in written form. Additionally, in some embodiments, the token processor(s) 116 may use the PAC token(s) 112(2) to generate the normalized text to include one or more punctuation marks and / or capitalization, and / or use the EOS / EOU token(s) 112(3) to generate the text to include one or more EOS symbols and / or one or more EOU symbols. While the example of Fig. 1A where the token processor(s) 116 is illustrated as separate from the machine learning model 108, in other examples the token processor(s) 116 may include a section of the machine learning model 108 (e.g., one or more encoders of the machine learning model 108).

[0076] Procedure 700 at block B708 ​​may include performing one or more operations using the normalized text. For example, the one or more operations may include presenting the normalized text using one or more client devices, further processing the normalized text using one or more additional processing units (e.g., one or more applications), and / or performing another type of operation.

[0077] Fig. Figure 8 illustrates a data flow diagram illustrating a method 800 for processing audio data using a machine learning model that performs ASR processing and EOS and / or EOU processing according to some embodiments of the present disclosure. At block B802, the method 800 may include generating, based on at least one machine learning model processing input data associated with speech, output data that is representative of one or more of first tokens associated with automatic speech recognition and one or more of second tokens associated with at least one or more sentence-end symbols or one or more utterance-end symbols.The machine learning model 108 can, for example, process the input data 106 (and / or the audio data 104) and, based on at least the processing, generate the output data 110, which represents the ASR token(s) 112(1) and the EOS / EOU token(s) 112(3). As described herein, in some examples the output data 110 can further represent the PAC token(s) 112(2) and / or the ITN token(s) 112(4). Additionally, in some examples, the output data 110 may represent one or more probabilities 114(1) associated with the ASR token(s) 112(1), and / or one or more probabilities 114(4) associated with the EOS / EOU token(s) 112(3).

[0078] Method 800 at block B804 can include generating text associated with language based on at least one or more first tokens and one or more second tokens, wherein the text includes at least one of the one or more sentence-end symbols or one or more utterance-end symbols. For example, processor(s) 116 can use ASR token(s) 112 and EOS / EOU token(s) 112(3) to generate the text (e.g., Unified Text), wherein the text includes the EOS symbol(s) and the EOU symbol(s). Additionally, in some embodiments, the token processor(s) 116 may use the PAC token(s) 112(2) to generate the text so that it includes one or more punctuation marks and / or capital letters, and / or use the ITN token(s) 112(4) to generate the text so that it includes the normalized format.

[0079] Procedure 800 at block B806 can include performing one or more operations on the text. For example, the one or more operations can include presenting the text using one or more client devices, further processing the text using one or more additional processing units (such as one or more applications), and / or performing another type of operation. EXEMPLARY LANGUAGE MODELS

[0080] In at least some embodiments, language models such as large language models (LLMs), vision language models (VLMs), multimodal language models (MMLMs), and / or other types of generative artificial intelligence (AI) can be implemented. These models can be capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, video, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., in USD format, such as OpenUSD), and / or the like, based on the context provided in prompts or queries. These language models can be considered "large" in embodiments based on the models being trained on massive datasets and architectures with a large number of learnable network parameters (weights and biases), such as millions or billions of parameters. The LLMs / VLMs / MMLMs / etc.They can be implemented for summarizing text data, analyzing and extracting insights from data (e.g., text, image, video data, etc.), and generating new text / images / videos, etc., in user-specified styles, nuances, and / or formats. The LLMs / VLMs / MMLMs / etc. of this disclosure can be used exclusively for text processing in some embodiments, while in other embodiments, multimodal LLMs can be implemented to accept, understand, and / or generate text and / or any other type of content, such as images, audio, 2D and / or 3D data (e.g., in USB formats), and / or video. Vision language models (VLMs) or, more generally, multimodal language models (MMLMs) can, for example, be implemented to process image, video, audio, text, 3D design (e.g.,to accept (in CAD) and / or other input data types and / or to generate or output image, video, audio, text, 3D design and / or other output data types.

[0081] Different types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented that use different techniques for understanding and / or generating outputs such as text, audio, video, image, 2D and / or 3D design or asset data, etc. In some embodiments, the LLM / VLM / MMLM / etc. architectures can use recurrent neural networks (RNNs) or long / short-term memory networks (LSTM networks), while in other embodiments, transformer architectures, such as those based on self-attention and / or cross-attention mechanisms (e.g., between context data and text data), can be used to understand and recognize relationships between words or tokens and / or context data (e.g., other text, video, image, design data, USD, etc.).One or more generative processing pipelines, including LLMs, VLMs, MMLMs, etc., may also include one or more diffusion blocks (e.g., noise suppressors). The LLMs, VLMs, MMLMs, etc., of this disclosure may include one or more encoder and / or decoder blocks. Discriminative or pure encoder models, such as BERT (Bidirectional Encoder Representations from Transformers), may be implemented for tasks involving language understanding, such as classification, sentiment analysis, and question answering, and may be referred to as entity recognition. As another example, generative or decoder-only models, such as GPC (Generative Pretrained Transformer), may be implemented for tasks involving language and content generation, such as text completion, story creation, and dialogue generation.Architectures that include both encoder and decoder components, such as the T5 (Text-to-Text Transformer), can be implemented to understand and generate content, for example, for translation and summarization. These examples are intended to be limiting, and any architecture type, including but not limited to those described herein, can be implemented depending on the specific implementation and the task(s) performed using the LLMs / VLMs / MMLMs / etc.

[0082] In various embodiments, LLMs / VLMs / MMLMs / etc. can be trained using unsupervised learning, where an LLM / VLM / MMLM / etc. learns patterns from large amounts of unlabeled text-audio-video-image-design-USD data. Due to comprehensive training, models in some embodiments may not require task-specific or domain-specific training. LLMs / VLMs / MMLMs / etc. that have undergone comprehensive pretraining on large amounts of unlabeled data can be referred to as foundation models and may be suitable for a variety of tasks, such as question-answering, summarizing, filling in missing information, translation, and image-video / design / USD / data generation. Some LLMs / VLMs / MMLMs / etc. may be tailored for a specific use case using techniques such as prompt tuning, fine-tuning, retrieval augmented generation (RAG), and adding adapters (e.g.,adapted neural networks and / or neural network layers that can tune or adjust input prompts to align the language model towards a particular task or domain), and / or the use of other fine-tuning or adaptation techniques that optimize the models for use in a particular task and / or within particular domains.

[0083] In some embodiments, the LLMSs / VLMSs / MMLMs / etc. of this disclosure can be implemented using various model alignment techniques. For example, in some embodiments, safeguards can be implemented to identify inappropriate or unwanted inputs (e.g., prompts) and / or outputs of the models. The system can use the safeguards and / or other model alignment techniques to either prevent a particular unwanted input from being processed using the LLMs / VLMs / MMLMs / etc., and / or to prevent the output or presentation (e.g., display, audio output, etc.) of information from being generated using the LLMs / VLMs / MMLMs / etc. In some embodiments, one or more additional models or layers thereof can be implemented to identify problems with the inputs and / or outputs of the models.These "protective measures" models can, for example, be trained to identify inputs and / or outputs that are "safe" or otherwise OK or desirable, and / or that are "unsafe" or otherwise undesirable for the particular application / implementation. Consequently, the LLMs / VLMs / MMLMs / etc. of this disclosure may be less likely to output speech / text / audio / design data / USD data, etc., that is offensive, vulgar, inappropriate, unsafe, out of scope, and / or otherwise undesirable for the particular application / implementation.

[0084] In some implementations, the LLMs / VLMs / etc. can be configured or enabled to access or use more plug-ins, application programming interfaces (APIs), databases, data stores, data pools, etc. For certain tasks or operations for which the model is not ideally suited, the model may, for example, have instructions (e.g., a training result and / or based on instructions in a given prompt) to access one or more plug-ins (e.g., third-party plug-ins) to assist in processing the current input. In such an example, where at least part of a prompt concerns restaurants or weather, the model may access one or more restaurant or weather plug-ins (e.g., via one or more APIs) to retrieve the relevant information.As another example, where at least part of an answer requires a mathematical calculation, the model can access one or more mathematical plugins or APIs to assist in solving the problem(s) and can then use the answer from the plugin and / or API in the model's output. This process can be repeated recursively over a number of iterations using any number of plugins and / or APIs until an answer to the input prompt can be generated that addresses any task / question / requirement / process / operation, etc. The model(s) can therefore draw not only on its own knowledge acquired through training on one or more large datasets, but also on the expertise and optimized nature of one or more external resources, such as APIs, plugins, and / or the like.

[0085] In some embodiments, multiple language models (e.g., LLMs / VLMs / MMLMs / etc.), multiple instances of the same language model, and / or multiple prompts served by the same language model or instance of the same language model can be implemented, executed, or accessed (e.g., using one or more plug-ins, user interfaces, APIs, databases, data stores, data pools, etc.) to provide output in response to the same query or in response to separate sections of a query. In at least one embodiment, multiple language models, e.g., language models with different architectures, language models trained on different (e.g., updated) datasets, can be served with the same input query and prompt (e.g., set of constraints, conditioners, etc.).In one or more embodiments, the language models can be different versions of the same Foundation model. In one or more embodiments, at least one language model can be instantiated as multiple agents; for example, more than one prompt can be provided to restrict, limit, or otherwise influence a style, content, character, etc., of the output provided. In one or more example, non-restrictive embodiments, the same language model can be instructed to provide output corresponding to a different role, perspective, character, or knowledge base, etc., as defined by a provided prompt.

[0086] In each such embodiment, the output of two or more (e.g., each) language models, of two or more versions of at least one language model, of two or more instantiated agents of at least one language model, and / or of two or more prompts provided to at least one language model can be further processed, for example, aggregated, compared, filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from a language model or version, instance, or agent can perhaps be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be prompted to produce or otherwise obtain output with respect to input source material, the output being associated with the input source material.Such an assignment might include, for example, the generation of a heading or text passage that is embedded (e.g., as metadata) in an input source text or image. In one or more embodiments, an output from a language model can be used to determine the validity of input source material for further processing or inclusion in a dataset. For example, a language model can be used to assess the presence (or absence) of a target word in a text passage or an object in an image, annotating the text or image to indicate such presence (or lack thereof). Alternatively, the determination from the language model can be used to determine whether the source material should be included in a carefully selected dataset, for example, without restriction.

[0087] Fig. Figure 9A is a block diagram of an exemplary generative language model system 900, suitable for use in implementing at least some embodiments of the present disclosure. In the Fig. The illustrated example 9A includes the generative language model system 900, a Retrieval Augmented Generation (RAG) component 992, an Input Processor 905, a Tokenizer 910, an Embed Component 920, Plug-ins / APIs 995, and a generative language model (LM) 930 (which may include an LLM, a VLM, a multimodal LM, etc.).

[0088] At a high level, the input processor 905 can receive an input 901 that includes text and / or other input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, Universal Scene Descriptor data (USD data – such as OpenUSD, etc.)), depending on the architecture of the generative LM 930 (e.g., LLM / VLM / MMLM / etc.). In some embodiments, the input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, the input 901 can include number sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., in tabular formats, JSON, or XML).In some implementations where the generative LM 930 is capable of processing multimodal inputs, the LM 930 input processor can combine (or omit) text with image data, audio data, video data, design data, USD data, and / or other types of input data, such as, but not limited to, those described herein. Using raw input text as an example, the LM 930 input processor can prepare raw input text in various ways. For instance, the LM 930 input processor can perform various types of text filtering to remove noise relevant to the text content (e.g., special characters, punctuation, HTML tags, stop words, parts of images, parts of audio files, etc.). In an example involving stop words (common words that tend to contain little semantic meaning), the LM 930 input processor can remove stop words to reduce noise and allow the generative LM 930 to focus on more meaningful content.The 905 input processor can normalize text, for example, by converting all characters to lowercase, removing accents, and / or handling special cases such as contractions or abbreviations to ensure consistency. These are just a few examples, and other types of input processing can be applied.

[0089] In such embodiments, a RAG component 992 (which may include one or more RAG models and / or be implemented using the generative LM 930 itself) can be used to retrieve additional information for use as part of the input 901 or prompt. RAG can be used to enhance the input to the LLM / VLM / MMLM / etc. with external knowledge, making answers to specific questions, retrievals, or requests more relevant, as in cases where specific knowledge is required. The RAG component 992 can retrieve this additional information (e.g., background information such as background text / images / videos / audio files / USD / CAD / etc.) from one or more external sources, which can then be passed along with the prompt to the LLM / VLM / MMLM / etc. to improve the accuracy of the model's answers or outputs.

[0090] In some embodiments, for example, the input 901 can be generated using the retrieval or input into the model (e.g., a question, a request, etc.) in addition to data retrieved using the RAG component 992. In some embodiments, the input processor 905 can analyze the input 901 and communicate with the RAG component 992 (or the RAG component 992 can be part of the input processor 905 in some embodiments) to identify relevant text and / or other data to provide to the generative LM 930 as additional context or information sources from which the reaction, response, or output 990 is generally to be identified.For example, if the input indicates that the user is interested in a desired tire pressure for a specific make and model of vehicle, the RAG component 992 can, using a RAG model that performs, for example, a vector search in an embedding space, retrieve the tire pressure information or corresponding text from the digital (embedded) version of the user manual for that specific make and model of vehicle. If a user revisits a chatbot in connection with a specific product offering or service, the RAG component 992 can retrieve a previously stored conversation history, or at least a summary thereof, and include the previous conversation history along with the current query / request as part of the input 901 to the generative LM 930.

[0091] The RAG component 992 can employ various RAG techniques. Naive RAG, for example, can be used for indexing, splitting documents into blocks, and applying it to an embedding model to generate embeddings that correspond to chunks. A user query can also be applied to the embedding model and / or another embedding model of the RAG component 992, and the chunk embeddings can be compared with the query's embeddings to identify the embeddings that are most similar to / correspond to the query. This information is then passed to the generative LM 930 to generate output.

[0092] In some implementations, more advanced RAG techniques can be used. For example, before passing chunks to the embedding model, the chunks can be subjected to pre-fetching processes (e.g., routing, rewriting, metadata analysis, expansion, etc.). Additionally, before generating the final embeddings, post-fetching processes (e.g., reordering, prompt compression, etc.) can be performed on the outputs of the embedding model for comparison with an input query.

[0093] Another example is modular RAG techniques, such as those similar to naive and / or advanced RAG, but which also include features like hybrid search, recursive retrieval and query engines, step-back approaches, subqueries, and hypothetical document embedding.

[0094] As another example, Graph RAG can use knowledge graphs as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information that is sent to the LLM / VLM / MMLM / etc. Instead of populating the model with chunks of data extracted from larger documents (or in addition to this, which can lead to a lack of context, factual accuracy, linguistic precision, etc.), Graph RAG can also provide the LLM / VLM / MMLM / etc. with structured entity information by combining the structured text description of the entity with its numerous properties and relationships, allowing the model to gain deeper insights. When implementing Graph RAG, the systems and procedures described herein use a graph as a content store, extract relevant chunks from documents, and request them from the LLM / VLM / MMLM / etc.to use them when answering questions. In such embodiments, the knowledge graph can contain relevant text content and metadata about the knowledge graph, as well as have an integrated vector database. In some embodiments, Graph RAG can use a graph as a subject matter expert, extracting relevant descriptions of concepts and entities for a query / prompt and passing them to the model as semantic context. These descriptions can include relationships between the concepts. In other examples, the graph can be used as a database, where part of a query / prompt can be mapped to a graph query, the graph query can be executed, and the LLM / VLM / MMLM / etc. can summarize the results.In such an example, the graph can store relevant factual information, and a query (natural language query) to a graph query tool (NL-to-graph query tool) as well as entity joining can be used. In some implementations, graph RAG (e.g., using a graph database) can be combined with standard RAG (e.g., vector database) and / or other RAG types to utilize multiple approaches.

[0095] In any embodiment, the RAG component 992 can implement a plug-in, API, user interface, and / or other functionality for performing RAG. For example, a graph RAG plug-in can be used by the LLM / VLM / MMLM / etc. to query the knowledge graph and extract relevant information for feeding into the model, and a standard or vector RAG plug-in can be used to query a vector database. The graph database can, for example, interact with the plug-in's REST interface in such a way that the graph database is decoupled from the vector database and / or the embedding models.

[0096] The Tokenizer 910 can segment text data into smaller units (tokens) for subsequent analysis and processing. Tokens can represent individual words, partial words, characters, or sections of audio, video, image, etc., depending on the implementation. Word-based tokenization divides the text into individual words, with each word treated as a separate token. Part-word tokenization distributes words into smaller, meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological variations and handle out-of-vocabulary words more effectively. Character-based tokenization represents each character as a separate token, allowing the generative LM 930 to process text at a fine-grained level. The choice of tokenization strategy may depend on factors such as the processing of the language, the task at hand, and / or characteristics of the training dataset.The Tokenizer 910 can therefore convert the (e.g. processed) text into a structured format according to a tokenization system implemented in the particular embodiment.

[0097] The Embedding Component 920 can use any known embedding technique to transform individual tokens into representations with semantic meaning (e.g., dense continuous vectors). For example, the Embedding Component 920 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot coding, term-frequency-inverse document-frequency coding (TF-IDF coding), one or more neural network embedding layers, and / or other methods.

[0098] In some implementations where the input 901 includes image data / video data / etc., the input processor 901 can resize the data to a standard size compatible with the format of a corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent display. The embedding component 920 can encode the image data using a known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where the input 901 includes audio data, the input processor 901 can resample an audio file to a consistent sampling rate for uniform processing, and the embedding component 920 can use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram).In some implementations where the input 901 includes video data, the input processor 901 can extract frames or resize them to extract the images, and the embedding component 920 can extract features such as optical flow embeddings or video embeddings and / or encode temporal information or image sequences. In some implementations where the input 901 includes multimodal data, the embedding component 920 can merge representations of different types of data (e.g., text, image, audio, USD, video, design, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.

[0099] The generative LM 930 and / or other components of the generative LM system 900 can use different types of neural network architectures depending on the implementation. Transformer-based architectures, such as those used in models like GPT, can be implemented and may include self-attention mechanisms that weight the importance of different words or tokens in the input sequence and / or feedforward networks that process the output of the self-attention layers, apply nonlinear transformations to the input representation, and extract higher-level features. Some non-restrictive example architectures include transformers (e.g.,Encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, crossmodal embedding models that learn shared embedding spaces, graph neural networks (GNNs), hybrid architectures that combine different types of architectures, adversarial networks such as generative adversarial networks or GANs, or adversarial autoencoders (AAEs) for shared distributional learning, and others. Therefore, depending on the implementation and architecture, the embedding component 920 can apply a coded representation of the input 901 to the generative LM 930, and the generative LM 930 can process the coded representation of the input 901 to produce an output 990 that may include response text and / or other types of data.

[0100] As described herein, in some embodiments the generative LM 930 may be configured to access or use plug-ins / APIs 995 (which may include one or more plug-ins, application programming interfaces (APIs), databases, data stores, data pools, etc.). For certain tasks or operations for which the generative LM 930 is not ideally suited, the model may have instructions (e.g., as a result of training and / or based on instructions in a given prompt, such as those retrieved using the RAG component 992) to access one or more plug-ins / APIs 995 (e.g., third-party plug-ins) to assist in processing the current input. In such an example, where at least part of the prompt concerns restaurants or weather, the model may access one or more restaurant or weather plug-ins (e.g.,(via one or more APIs), at least part of the input request regarding the specific plugin / API 995 is sent to the plugin / API 995, the plugin / API 995 can process the information and send a response back to the generative LM 930, and the generative LM 930 can use the response to generate the output 990. This process can, for example, be recursively performed over a number of iterations using any number of plugin / APIs 995 until an output 990 is generated that handles every request / question / request / process / operation / etc. from the input 901. The model(s) can therefore not only rely on its own knowledge from training on one or more large datasets and / or on data retrieved using the RAG component 992, but also on expertise and optimized nature of one or more external resources, such as the plug-ins / APIs 995.

[0101] Fig. Figure 9B is a block diagram of an example implementation where the generative LM 930 includes a transformer-encoder-decoder. For example, assume that input text such as "Who discovered gravity" (e.g., through the Tokenizer 910 of the LM 930) is used to create a 930 Transformer-Encoder-Decoder. Fig. 9A) is tokenized into tokens such as words, and each token (e.g., through the embedding component 920 of the Fig. 9A) is encoded into a corresponding embedding (e.g., with size 512). Since this token typically does not represent the token's position in the input sequence, any known technique can be used to add positional encoding to each token embedding to encode the sequential relationship and context of the tokens in the input sequence. Therefore, the resulting embeddings can be applied to one or more Encoders 935 of the generative LM 930.

[0102] In an exemplary implementation, the Encoder 935 forms an encoder stack, with each encoder including a self-attention layer and a feedforward network. In an exemplary transformer architecture, each token (e.g., each word) flows through a separate path. Each encoder can therefore accept a sequence of vectors that take each vector through the self-attention layer, then the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used.For example, to calculate a self-attention value for each token (word), a query vector, a key vector, and a value vector can be created for each token. A self-attention value can be calculated for token pairs by taking the dot product of the query vector with the corresponding key vectors, normalizing the resulting values, multiplying them by the corresponding value vectors, and summing the weighted value vectors. The encoder can apply multi-headed attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector that encodes the input. An attention projection layer 940 can convert the context vector into attention vectors (keys and values) for the decoder(s) 945.

[0103] In an exemplary implementation, the Decoder(s) 945 form a Decoder stack, with each Decoder including a Self-Attention layer, an Encoder-Decoder Self-Attention layer that uses the attention vectors (keys and values) from the Encoder to focus on relevant parts of the input sequence, and a feedforward network. As with the Encoder(s) 935, in an exemplary Transformer architecture, each token (e.g., word) flows through a separate path into the Decoder(s) 945. During a first pass, the Decoder(s) 945, a Classifier 950, and a Generating Mechanism 955 can generate an initial token, and the Generating Mechanism 955 can apply the generated token as an input during a second pass. The process can repeat in a loop by successively generating tokens (e.g., words, words, words, etc.).Words) are generated and added to the output from the previous pass, and the token embeddings of the composite sequence with positional codes are applied as an input to the decoder(s) 945 during a subsequent pass, generating one token at a time (known as autoregression) until a character or token representing the end of the response is predicted. Within a decoder, the self-attention layer is typically restricted to dealing only with previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In an exemplary implementation, the encoder-decoder-attention layer works similarly to (e.g., multi-headed) self-attention in the encoder(s) 935, except that it establishes its fetches from the layer below and the keys and values ​​(e.g.,Matrix) from the output of the encoder 935.

[0104] The Decoder(s) 945 can therefore output a somewhat decoded (e.g., vector) representation of the input applied during a given pass. The Classifier 950 can include a multi-class classifier comprising one or more neural network layers that project the decoded (e.g., vector) representation into an appropriate dimensionality (e.g., one dimension for each supported word or token in the output vocabulary), as well as a softmax operation that converts logits into probabilities. The Generation Mechanism 955 can therefore select or extract a word or token based on an appropriate predicted probability (e.g., selecting the word with the highest predicted probability) and append it to the output from a previous pass, so that each word or token is generated sequentially.The 955 generation mechanism can repeat the process, triggering successive decoder inputs and corresponding predictions, until a character or token is selected or sampled that represents the end of the response, at which point the 955 generation mechanism can output the generated response.

[0105] Fig. 9C is a block diagram of an exemplary implementation where the generative LM 930 incorporates a decoder-transformer-only architecture. The Decoder 960 of the Fig. 9C decoders, for example, can work similarly to the 945 decoder(s). Fig. 9B, with the exception that each decoder 960 of the Fig. 9C omits the encoder-decoder self-attention layer (since there is no encoder in this implementation). The Decoder 960(s) therefore form a decoder stack, in which each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a character or token representing the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (which corresponds, for example, to embeddings with positional encodings) can be applied to the Decoder 960(s). As with the Decoder 945(s), Fig. In 9B, each token (e.g., each word) can flow through a separate path to the decoder(s) 960, and the decoder(s), a classifier 965, and a generation mechanism 970 can use autoregression to sequentially generate one token at a time until a character or token is predicted that represents the end of the response. The classifier 965 and the generation mechanism 970 can perform similar work to the classifier 950 and the generation mechanism 955 of the Fig. 9B, wherein the generation mechanism 970 selects or samples each successive output token based on a corresponding predicted probability and appends it to the output from a previous pass, each token being generated sequentially until a character or token representing the end of the response is selected or sampled. These and other architectures described herein are merely examples, and other suitable architectures may be implemented within the scope of this disclosure. EXAMPLE CALCULATION DEVICE

[0106] Fig. Figure 10 is a block diagram of an exemplary computing device(s) 1000 suitable for use in implementing some embodiments of the present disclosure. The computing device 1000 can include a connection system 1002 that directly or indirectly couples the following devices: main memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output ports (I / O ports) 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., display(s)), and one or more logic units 1020. In at least one embodiment, the one or more computing devices 1000 can comprise one or more virtual machines (VMs), and / or each of the components thereof can comprise virtual components (e.g., virtual hardware components).As non-restrictive examples, one or more of the GPUs 1008 can comprise one or more vGPUs, one or more of the CPUs 1006 can comprise one or more vCPUs, and / or one or more of the logic units 1020 can comprise one or more virtual logic units. Thus, one or more computing devices 1000 can include discrete components (e.g., a complete GPU allocated to computing device 1000), virtual components (e.g., a portion of a GPU allocated to computing device 1000), or a combination thereof.

[0107] Although the various blocks of the Fig. Where components 10 are shown connected by lines via the connection system 1002, this is not intended as a limitation and is for clarity only. In some embodiments, for example, a presentation component 1018, such as a display device, may be considered an I / O component 1014 (if, for example, the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 may include memory (for example, the memory 1004 may represent a storage device in addition to the memory of the GPUs 1008, the CPUs 1006, and / or other components). In other words, the computing device of Fig. Section 10 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other types of device or system, as all are considered to be within the protective scope of the computing device. Fig. 10 will be considered.

[0108] The connection system 1002 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The connection system 1002 can include one or more bus or link types, such as an Industry Standard Architecture bus (ISA bus), an Extended Industry Standard Architecture bus (EISA bus), a Video Electronics Standards Association bus (VESA bus), a Peripheral Component Interconnect bus (PCI bus), a Peripheral Component Interconnect Express bus (PCIe bus), and / or any other type of bus or link. In some embodiments, there are direct connections between components. For example, the CPU 1006 can be directly connected to the main memory 1004. Furthermore, the CPU 1006 can be directly connected to the GPU 1008. In the case of a direct or point-to-point connection between components, the 1002 connection system can include a PCIe link to establish the connection.In these examples, a PCI bus does not need to be included in the computing device 1000.

[0109] The working memory 1004 can include a variety of computer-readable media. These computer-readable media can be any available media that the computing device 1000 can access. They can include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, computer-readable media can include computer storage media and communication media.

[0110] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory can store 1004 computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system).Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, Digital Versatile Discs (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that the computing device 1000 can access. As used herein, computer storage media do not include signals as such.

[0111] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any media for transmitting information. The term "modulated data signal" can refer to a signal in which one or more properties are set or modified such that information is encoded in the signal. Computer storage media can include, by way of example and without limitation, wired media, such as a wired network or a directly wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of the above should also be included within the scope of protection of computer-readable media.

[0112] The CPU(s) 1006 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the procedures and / or processes described herein. The CPU(s) 1006 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a variety of software threads concurrently. The CPU(s) 1006 can include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1000, the processor can be, for example, an Advanced RISC Machines (ARM) processor implemented using a Reduced Instruction Set Computing (RISC) processor, or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1000 can include one or more CPUs 1006 in addition to one or more microprocessors or supplementary coprocessors, such as mathematical coprocessors.

[0113] In addition to or as an alternative to the one or more CPUs 1006, the one or more GPUs 1008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the procedures and / or processes described herein. One or more of the GPUs 1008 may be an integrated GPU (e.g., with one or more of the CPUs 1006), and / or one or more of the GPUs 1008 may be a separate GPU. In embodiments, one or more of the GPUs 1008 may be a coprocessor of one or more of the CPUs 1006. The GPUs 1008 may be used by the computing device 1000 to render graphics (e.g., 3D graphics) or to perform general-purpose calculations. For example, the GPU(s) 1008 can be used for general-purpose computing on GPUs (GPGPU).The GPU(s) 1008 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU(s) 1008 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 1006 received via a host interface). The GPU(s) 1008 can include graphics memory, such as display memory for storing pixel data or other useful data, such as GPGPU data. The display memory can be included as part of the main memory 1004. The GPU(s) 1008 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or connect them via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1008 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can have its own dedicated memory or share memory with other GPUs.

[0114] In addition to or as an alternative to the CPU(s) 1006 and / or the GPU(s) 1008, the one or more logic unit(s) 1020 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1006, the GPU(s) 1008, and / or the logic unit(s) 1020 may, separately or jointly, perform any combination of methods, processes, and / or parts thereof. One or more of the logic units 1020 may be part of and / or integrated into one or more of the CPUs 1006 and / or one or more of the GPUs 1008, and / or one or more of the logic units 1020 may be discrete components or otherwise separate from the CPUs 1006 and / or the GPUs 1008.In embodiments, one or more logic units 1020 can be a coprocessor of one or more of the CPU(s) 1006 and / or one or more of the GPU(s) 1008.

[0115] Examples of one or more logic units 1020 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), and application-specific integrated circuits. (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output elements (I / O elements), peripheral component interconnect (PCI) or PCI Express elements (PCIe elements), and / or the like.

[0116] The communication interface 1010 can include one or more receivers, transmitters, and / or transceivers that enable the computing device 1000 to communicate with other computing devices over an electronic communication network, including wired and / or wireless communications. The communication interface 1010 can include components and functionality to enable communication over a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 1020 and / or the communication interface 1010 can include one or more data processing units (DPUs) to transfer data received via a network and / or the connection system 1002 directly to one or more GPUs 1008 (e.g., a memory thereof).

[0117] The I / O ports 1012 enable the computing device 1000 to be logically coupled with other devices, including the I / O components 1014, the one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated with) the computing device 1000. Illustrative I / O components 1014 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1014 can provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs can be transmitted to a suitable network element for further processing.A NUI can implement any combination of speech capture, stylus capture, facial recognition, biometric recognition, gesture recognition (both on-screen and off-screen), air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the Computing Device 1000. The Computing Device 1000 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the Computing Device 1000 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the outputs from the accelerometers and gyroscopes can be used by the Computing Device 1000 to render immersive augmented reality or virtual reality.

[0118] The power supply 1016 can include a wired power supply, a battery power supply, or a combination of both. The power supply 1016 can provide power to the computing device 1000 to enable the functioning of the computing device 1000's components.

[0119] The presentation component(s) 1018 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component(s) 1018 can receive data from other components (e.g., the one or more GPU(s) 1008, CPU(s) 1006, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). EXEMPLARY DATA CENTER

[0120] Fig. Figure 11 illustrates an exemplary data center 1100 that can be used in at least one embodiment of the present disclosure. The data center 1100 can include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0121] As in Fig. As shown in Figure 11, the data center infrastructure layer 1110 can include a resource orchestrator 1112, clustered compute resources 1114, and node compute resources (“node RRs”) 1116(1)-1116(N), where “N” represents any positive integer. In at least one embodiment, the node RRs 1116(1)-1116(N) can include, but are not limited to, a number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules, etc. In some embodiments, one or more node RRs of the node RR can beNode RRs 1116(1)-1116(N) correspond to a server that has one or more of the computing resources mentioned above. Additionally, in some embodiments, the node RRs 1116(1)-1116(N) may include one or more virtual components, such as vGPUs, vCPUs and / or the like, and / or one or more of the node RRs 1116(1)-1016(N) may correspond to a virtual machine (VM).

[0122] In at least one embodiment, grouped compute resources 1114 can include separate groupings of node RRs 1116 housed in one or more racks (not shown), or many racks housed in data centers at different geographic locations (also not shown). Separate groupings of node RRs 1116 within grouped compute resources 1114 can include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RRs 1116, including the CPUs, GPUs, DPUs, and / or other processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.The one or more racks can also include any number of power modules, cooling modules and / or network switches in any combination.

[0123] The resource orchestrator 1112 can configure or otherwise control one or more node RRs 1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, the resource orchestrator 1112 can include a software design infrastructure management unit (SDI management unit) for the data center 1100. The resource orchestrator 1112 can include hardware or software, or a combination thereof.

[0124] In at least one embodiment, as in Fig. As shown in Figure 11, the framework layer 1120 can include a job scheduler 1128, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 can include a framework to support software 1132 of software layer 1130 and / or one or more applications 1142 of application layer 1140. The software 1132 or the application(s) 1142 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 can be a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter referred to as "Spark"), which can utilize a distributed file system 1138 for processing large amounts of data (e.g., "Big Data") without being limited to it.In at least one embodiment, the job scheduler 1128 can include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1100. The configuration manager 1134 can be able to configure different layers, such as the software layer 1130 and the framework layer 1120, including Spark and the distributed file system 1138 to support large-scale data processing. The resource manager 1136 can be able to manage clustered or grouped compute resources that are mapped to or associated with the distributed file system 1138 and the job scheduler 1128 for support. In at least one embodiment, the clustered or grouped compute resources can include the grouped compute resources 1114 on the data center infrastructure layer 1110.The resource manager 1136 can coordinate with the resource orchestrator 1112 to manage these mapped or allocated computing resources.

[0125] In at least one embodiment, the software 1132, which is enclosed in the software layer 1130, may include software that is used by at least sections of the node RRs 1116(1)-1116(N), grouped computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0126] In at least one embodiment, the application(s) 1142 included in the application layer 1140 may include one or more types of applications used by at least sections of the node RRs 1116(1)-1116(N), grouped compute resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may, but are not limited to, include any number of genome applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0127] In at least one embodiment, the configuration manager 1134, the resource manager 1136, and / or the resource orchestrator 1112 can implement any number and any type of self-modifying actions based on any set and any type of data acquired in any technically feasible way. Self-modifying actions can relieve a data center operator of the data center 1100 of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly functioning parts of a data center with substandard performance.

[0128] The Data Center 1100 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture, using software and / or computing resources described above with reference to Data Center 1100.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to derive or predict information using resources described above with reference to Computing Center 1100, using weighting parameters computed by one or more training techniques such as those described herein.

[0129] In at least one embodiment, the data center can use 1100 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above can be configured as a service to allow users to perform training or inference on information, such as image recognition, speech recognition, or other AI services. EXEMPLARY NETWORK ENVIRONMENTS

[0130] Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or device types. The client devices, servers, and / or other device types (e.g., each device) may be connected to one or more instances of the one or more computing devices. Fig. 10. Each device may include similar components, features, and / or functionality of one or more computing devices 1000. If backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may additionally be included as part of a data center 1100, an example of which is described in more detail herein with reference to Fig. 11 is described.

[0131] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can include multiple networks or a network of networks. For example, the network can include one or more Wide Area Networks (WANs), one or more Local Area Networks (LANs), one or more public networks, such as the Internet, and / or public networks, such as a Public Switched Telephone Network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0132] Compatible network environments can include one or more peer-to-peer network environments, where a server may not be included in a network environment, and one or more client-server network environments, where one or more servers may be included in a network environment. In peer-to-peer network environments, the functionality described herein can be implemented with reference to one or more servers on any number of client devices.

[0133] In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer can include a framework for supporting software of a software layer and / or one or more applications of the application layer. The software or application(s) can each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be a type of free and open-source software web application framework that uses, for example, a distributed file system for processing large amounts of data (e.g., "Big Data"), but is not limited to this.

[0134] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination (or one or more parts) of the computing and / or data storage functions described herein. Any of these various functions can be distributed across multiple locations from central or core servers (e.g., from one or more data centers distributed across a state, region, country, the Earth, etc.). If a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can allocate at least one part of the functionality to the edge server(s). A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0135] The client device(s) may incorporate at least some of the components, features, and functionality of the exemplary computing device(s) 1000 described herein with reference to Fig.10. For example, and without limitation, a client device may be a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a flying vehicle, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment device, a vehicle computing system, an embedded system controller, a household appliance, a consumer electronics system, a workstation, an edge device, any combination of these devices, or any other suitable device.

[0136] The disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules that are executed by a computer or other machine, such as a personal data assistant or other handheld device. In general, program modules, which include routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements particular abstract data types. The disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The disclosure can also be implemented in distributed computing environments, in which tasks are performed by remote processing devices linked by a communication network.

[0137] As used herein, any mention of "and / or" in relation to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" can mean only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Additionally, "at least one of element A or element B" can include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, "at least one element A and element B" can include at least one element A, at least one element B, or at least one element A and at least one element B.

[0138] The subject matter of this disclosure is described herein with a certain degree of accuracy to comply with legal requirements. However, the description itself is not intended to limit the scope of protection afforded by this disclosure. Instead, the inventors have considered that the claimed subject matter could also be embodied in other types, so that it includes different steps or combinations of steps similar to those described in this patent specification, in conjunction with other present or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to denote different elements of methods used, the terms should not be interpreted as a specific sequence among or between the various steps disclosed herein, except and exclusively where the sequence of individual steps is explicitly described. EXAMPLE PARAGRAPHS

[0139] The disclosure of this application also includes the following exemplary paragraphs: A: A method comprising: generating audio embeds associated with audio data representative of user speech; generating, based on at least one machine learning model processing input data associated with the audio embeds, output data representative of tokens associated with user speech, wherein at least one first token is associated with automatic speech recognition and at least one second token is associated with inverse text normalization; generating, based on at least the tokens, normalized text representing user speech; and performing one or more operations using the normalized text. B: Method according to paragraph A, wherein: at least one third part of the tokens is assigned to at least one of a sentence end or an utterance end; and the normalized text includes at least one of a first indication of the sentence end or a second indication of the sentence end. C: Method according to either paragraph A or paragraph B, wherein: at least one third of the tokens is assigned to at least one of one or more punctuation marks or one or more capital letters; and the normalized text includes the at least one of the one or more punctuation marks or the at least one or more capital letters. D: Method according to any of paragraphs AC, wherein: the output data further represent probabilities assigned to the tokens; and the generation of the normalized text representing the user language furthermore is based on at least the probabilities. E: Method according to one of paragraph AD, wherein: at least the first part of the tokens assigned to automatic speech recognition includes at least one of the following: one or more of first tokens representing one or more letters; one or more of second tokens representing one or more sections of one or more first words; or one or more of third tokens representing one or more second words; and at least the second part of the tokens assigned to inverse text normalization includes at least one of the following: one or more of fourth tokens representing one or more numbers; or one or more of fifth tokens representing one or more symbols assigned to one or more of third words. F: Method according to any of paragraphs AE, wherein the tokens are associated with one or more of the first frames of the audio data, and wherein the method further comprises: generating, based on at least the machine learning model processing the input data, second output data representative of second tokens associated with the user language, wherein at least a first part of the second tokens is associated with automatic speech recognition, and at least a second part of the tokens is associated with inverse text normalization, wherein the generation of the normalized text representing the user language is further based on at least the second tokens. G: Method according to any of paragraphs AF, wherein performing the one or more operations using the normalized text includes at least one of the following: causing at least one section of the normalized text to be processed using one or more second machine learning models; or causing the presentation of an output that is associated with at least one section of the normalized text. H: A method according to any of paragraphs AG, further comprising: obtaining second audio data representative of the second user language and ground-truth data representative of one or more second tokens associated with the second user language, wherein at least a portion of the one or more second tokens is associated with inverse text normalization; generating one or more second audio embeds associated with the second audio data; generating, based at least on the machine learning model processing second input data associated with the one or more second audio embeds, second output data representative of one or more third tokens associated with the second user language;and updating one or more parameters associated with the machine learning model, based on at least one or more third tokens and one or more second tokens. I: System comprising: one or more processors for: generating, based on at least one machine learning model processing input data associated with user language, output data representative of: one or more of first tokens representing one or more letters; and one or more of second tokens representing at least one or more numbers or one or more symbols representing one or more words; generating, based on at least one or more of the first tokens and one or more of the second tokens, text representing the user language and including one or more numbers or one or more symbols; and performing one or more operations using the text. J: System according to paragraph I, wherein the one or more processors are furthermore, based on at least audio data representative of the user language, to generate input data representative of one or more embeddings corresponding to the one or more frames associated with the audio data. K: System either according to paragraph I or paragraph J, wherein: the one or more second tokens are assigned to inverse text normalization; and the text includes normalized text that corresponds to the user language. L: System according to one of paragraphs IK, wherein: the output data further represents one or more of third tokens that are associated with at least one of a sentence end or an utterance end; and the text representing the user language is further generated based on at least one or more of third tokens and includes at least a first indication of the sentence end or a second indication of the utterance end. M: System according to any of paragraphs IL, wherein: the output data further represents one or more third tokens associated with one or more punctuation marks; and the text representing the user language is further generated based on at least one or more third tokens and includes one or more punctuation marks. N: System according to one of paragraphs IM, wherein: the output data are furthermore representative of one or more first probabilities associated with the one or more of first tokens, and of one or more second probabilities associated with the one or more of second tokens; and the text representing the user language is furthermore generated based on at least the one or more first probabilities and the one or more second probabilities. O: System according to one of paragraphs IN, wherein: the output data is associated with one or more of first frames of audio data representing the user language; the one or more processors are furthermore, based at least on the machine learning model processing the input data, to generate second output data associated with one or more second frames of the audio data, wherein the second output data is representative of one or more of third tokens that are representative of at least one of the one or more numbers or of the one or more symbols that represent the one or more words; and the text is furthermore generated based on at least one or more of third tokens. P: System according to one of the paragraphs IO, wherein performing the one or more operations using the normalized text includes at least one of the following: causing at least one section of the text to be processed using one or more second machine learning models; or causing the presentation of an output associated with at least one section of the text. Q: System according to any of the paragraphs IP, wherein the machine learning model includes at least the following: one or more encoders for processing the audio data to generate the input data; and one or more decoders for processing the input data to generate the output data that is representative of one or more of the first tokens and one or more of the second tokens. R: System according to any of paragraphs IQ, wherein the system consists of at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twinning operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs);A system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more Multimodal Language Models (LLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one type of augmented reality content, virtual reality content, or mixed reality content; systems for implementing one or more multimodal language models; systems that use or employ one or more inference microservices; systems that employ one or more machine learning models in a service or microservice together with an OS-level virtualization package (e.g., a container); a system that contains one or more virtual machines (VMs); a system that is at least partially deployed in a data center;or a system that is implemented at least partially using cloud computing resources. S: One or more processors comprising: processing circuit arrangements for generating normalized text mapped to user speech based on at least one or more tokens, wherein the one or more tokens are generated based on at least the following: an encoder associated with a machine learning model processing audio data representative of the user speech to produce an initial output; and a decoder associated with the machine learning model processing the initial output to produce a second output representative of the one or more tokens. T: One or more processors according to paragraph S, wherein the machine learning model is deployed as an inference microservice that includes the machine learning model and an operating system-level (OS-level) virtualization package, wherein the OS-level virtualization package includes software for running the machine learning model and enterprise management software for performing one or more telemetry operations with respect to the machine learning model.

[0140] It is understood that the aspects and embodiments described above are purely exemplary, and that modifications of details may be made within the scope of protection of the claims.

[0141] Each device, each method and each feature disclosed in the description, and (where applicable) the claims and drawings, may be provided independently or in any suitable combination.

[0142] Reference numerals appearing in the claims are for illustrative purposes only and do not restrict the scope of protection of the claims.

Claims

[1] Procedure comprising the following: Generating audio embeds that are associated with audio data representative of user speech; Generating, based on at least one machine learning model that processes input data associated with audio embeddings, output data representative of tokens associated with user language, wherein at least one first part of the tokens is associated with automatic speech recognition, and at least one second part of the tokens is associated with inverse text normalization; Generate based on at least the tokens of normalized text representing the user language; and Performing one or more operations using the normalized text. [2] Method according to claim 1, wherein: at least one third of the tokens is assigned to at least one of the ends of a sentence or utterance; and The normalized text includes at least one indication of the end of a sentence or a second indication of the end of an utterance. [3] Method according to any one of the preceding claims, wherein: at least one third of the tokens is assigned to at least one of one or more punctuation marks or one or more capital letters; and the normalized text that includes at least one of the following punctuation marks or at least one of the following capital letters. [4] Method according to any one of the preceding claims, wherein: the output data also represent probabilities assigned to the tokens; and the generation of the normalized text representing the user language, furthermore based at least on the probabilities. [5] Method according to any one of the preceding claims, wherein: At least the first part of the tokens, which is assigned to automatic speech recognition, includes at least one of the following: one or more of the first tokens representing one or more letters; one or more second tokens representing one or more sections of one or more first words; or one or more third tokens representing one or more second words; and At least the second part of the tokens, which is assigned to inverse text normalization, includes at least one of the following: one or more of the fourth tokens representing one or more numbers; or one or more fifth tokens representing one or more symbols associated with one or more third words. [6] Method according to any one of the preceding claims, wherein the tokens are assigned to one or more of the first frames of the audio data, and wherein the method further comprises: Generating, based on at least the machine learning model that processes the input data, second output data that are representative of second tokens associated with the user language, wherein at least a first part of the second tokens is associated with automatic speech recognition, and at least a second part of the tokens is associated with inverse text normalization, where the generation of the normalized text representing the user language is furthermore based on at least the second token. [7] A method according to any one of the preceding claims, wherein performing the one or more operations using the normalized text comprises at least one of the following: To ensure that at least one section of the normalized text is processed using one or more secondary machine learning models; or Initiating the presentation of an output that is associated with at least one section of the normalized text. [8] A method according to any of the foregoing claims, further comprising: Obtaining second audio data representative of the second user language and ground truth data representative of one or more second tokens associated with the second user language, wherein at least a portion of the one or more second tokens is associated with inverse text normalization; Creating one or more secondary audio embeds that are associated with the secondary audio data; Generate, based on at least the machine learning model processing second input data associated with one or more second audio embeddings, second output data representative of one or more third tokens associated with the second user language; and Updating one or more parameters associated with the machine learning model, based on at least one or more third tokens and one or more second tokens. [9] System comprising the following: one or more processors for: Generate output data representative of the following, based on at least one machine learning model that processes input data mapped to user language: one or more of the first tokens, representing one or more letters; and one or more of second tokens that are representative of at least one or more numbers or one or more symbols that represent one or more words; Generate, based on at least one or more first tokens and one or more second tokens, text representing the user language and including one or more numbers or one or more symbols; and Performing one or more operations using the text. [10] System according to claim 9, wherein the one or more processors are further said to generate, based at least on audio data representative of the user language, the input data representative of one or more embeddings corresponding to the one or more frames associated with the audio data. [11] System according to one of claims 9 or 10, wherein: that one or more second tokens are assigned to inverse text normalization; and The text includes normalized text that corresponds to the user's language. [12] System according to any one of claims 9 to 11, wherein: the output data furthermore represent one or more third-party tokens that are associated with at least one sentence ending or utterance ending; and the text representing the user language is furthermore generated based on at least one or more third tokens and includes at least a first indication of the sentence end or a second indication of the utterance end. [13] System according to any one of claims 9 to 12, wherein: System according to any one of claims 9 or 11, wherein: the output data may also represent one or more third-party tokens associated with one or more punctuation marks; and the text representing the user language, which is furthermore generated based on at least one or more third-party tokens and includes one or more punctuation marks. [14] System according to any one of claims 9 to 13, wherein: the output data are furthermore representative of one or more first probabilities associated with one or more first tokens, and of one or more second probabilities associated with one or more second tokens; and the text representing the user language is furthermore generated based on at least one or more first probabilities and one or more second probabilities. [15] System according to any one of claims 9 to 14, wherein: the output data is associated with one or more of the first frames of audio data representing the user's speech; furthermore, the one or more processors, based at least on the fact that the machine learning model processes the input data, are to generate second output data that is associated with one or more second frames of the audio data, wherein the second output data is representative of one or more third tokens that are representative of at least one of the one or more numbers or of the one or more symbols that represent the one or more words; and The text is furthermore generated based on at least one or more third-party tokens. [16] System according to any one of claims 9 to 15, wherein performing the one or more operations using the normalized text comprises at least one of the following: To ensure that at least one section of the text is processed using one or more secondary machine learning models; or Initiating the presentation of an output that is associated with at least one section of the text. [17] System according to any one of claims 9 to 16, wherein the machine learning model includes at least the following: one or more encoders from processing the audio data to generate the input data; and one or more decoders to process the input data in order to generate the output data that is representative of one or more of the first tokens and one or more of the second tokens. [18] System according to any one of claims 9 to 17, wherein the system consists of at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more operations for digital twinning; a system for performing light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system that is implemented using an edge device; a system that is implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more Vision Language Models (VLMs); a system for performing operations using one or more multimodal language models (LLMs); a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one type of augmented reality content, virtual reality content, or mixed reality content; Systems for implementing one or more multimodal language models; Systems that use or employ one or more inference microservices; Systems that use one or more machine learning models in a service or microservice together with an OS-level virtualization package (e.g., a container); a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [19] One or more processors comprising the following: Processing circuit arrangements for generating normalized text mapped to user language, based on at least one or more tokens, wherein the one or more tokens are generated based on at least the following: an encoder that is assigned to a machine learning model processing audio data representative of the user's speech to generate an initial output; and a decoder associated with the machine learning model that processes the first output to generate a second output that is representative of the one or more tokens. [20] One or more processors according to claim 19, wherein the machine learning model is used as an inference microservice that includes the machine learning model and an operating system-level (OS-level) virtualization package, wherein the OS-level virtualization package includes software for running the machine learning model and enterprise management software for performing one or more telemetry operations with respect to the machine learning model.

Images