Generating unified text using speech recognition models for conversational AI systems and applications

Abstract

The invention relates to generating unified text using speech recognition models for conversational AI systems and applications. In examples, generating unified text for AI systems and applications using a speech recognition model is described herein. Systems and methods are disclosed for generating unified text related to user speech using a trained machine learning model, wherein the unified text includes punctuation marks, word capitalization, inverse text normalization formats, sentence end (EOS) detection, and / or utterance end (EOU) detection. For example, a machine learning model may receive, as input, audio data representing speech. The machine learning model may then process the audio data and generate output data related to the speech based at least on the processing procedure. In some examples, the output data may represent markings, such as markings related to automatic speech recognition processing, punctuation and capitalization processing, EOS and / or EOU processing, and / or inverse text normalization processing. In these examples, the tokens may be processed to generate unified text.

Description

Background Technology

[0001] Automatic speech recognition (ASR) systems are used to process a user's speech, converting it into text. However, the raw text output by an ASR system is in spoken form and lacks reliability, for example, using colloquial words (e.g., writing "$100" as "one hundred dollars"), missing capital letters, and / or punctuation. Therefore, additional systems are often needed to further process the raw text, such as systems that insert punctuation and / or correct the text to capitalization, and / or systems that convert spoken words to written form (e.g., perform inverse text normalization). Furthermore, in some cases, such as when the text is used by one or more downstream applications (e.g., natural language understanding applications and / or machine translation applications), the text may be additionally processed using systems that identify speech-related sentence endings and / or utterance endings.

[0002] However, problems can arise when using an architecture that incorporates multiple systems to process audio data and / or text to generate the final text format. For example, each system may be prone to output errors, such as the ASR system outputting text containing incorrect words. Furthermore, because these systems operate sequentially, if the initial system in the sequence (e.g., the ASR system) outputs an error, that error can propagate to the remaining systems, leading to a degraded output quality. Additionally, because the systems operate sequentially by processing the output from previous systems, the overall latency of the architecture increases with the number of systems used to process the audio data. For example, each system may have its own processing latency, where the overall latency of the architecture could be the sum of the latencies of all systems. Summary of the Invention

[0003] Embodiments of this disclosure relate to generating unified text for AI systems and applications using speech recognition models. Systems and methods using machine learning models trained to generate unified text associated with user speech are disclosed, wherein the unified text includes punctuation, word capitalization, inverse text normalization (ITN) formatting, sentence end (EOS) detection, and / or utterance end (EOU) detection. For example, the machine learning model may receive audio data representing user speech as input. In some examples, the audio data undergoes initial processing before being input to the machine learning model, such as generating input data representing embeddings associated with audio data frames. The machine learning model can then process the audio data and / or the input data and generate output data associated with unified text, at least based on this processing. In some examples, the output data may represent tokens, such as tokens associated with Automatic Speech Recognition (ASR) processing, Punctuation and Capitalization (PAC) processing, EOS processing, EOU processing, and / or ITN processing. In these examples, these tokens can be processed to generate unified text.

[0004] Compared to traditional systems, the system disclosed herein can use a single machine learning model to generate output data representing uniform text, which includes punctuation, word capitalization, ITN formatting, EOS detection, and / or EOU detection. Therefore, the system disclosed herein can reduce the number of systems, models, modules, applications, and / or other processing components required to generate such text in a final format for use by other applications and / or users. As described herein, by reducing the number of processing components, the system disclosed herein can also reduce the number of processing-related errors, as it is a single model prone to outputting errors, rather than multiple processing components, and / or can reduce the overall latency associated with processing. Attached Figure Description

[0005] The following will describe in detail, with reference to the accompanying drawings, the system and method for generating unified text using a speech recognition model for AI systems and applications, wherein:

[0006] Figure 1A Examples of a process for generating unified text using a machine learning model according to some embodiments of the present disclosure are shown;

[0007] Figure 1B Examples of machine learning models that can be trained to generate unified text according to some embodiments of the present disclosure are shown;

[0008] Figure 2 Examples of generating a tag set for an audio data frame according to some embodiments of this disclosure are shown;

[0009] Figure 3Examples of techniques for selecting tags associated with frames and / or groups of frames, according to some embodiments of this disclosure, are shown;

[0010] Figures 4A-4B Examples of different types of processing that can be performed by a machine learning model according to some embodiments of this disclosure are shown;

[0011] Figure 5A A data flow diagram illustrating a process for training a machine learning model to perform various types of processing, according to some embodiments of the present disclosure, is shown.

[0012] Figure 5B Examples of generating ground truth data for training machine learning models are shown according to some embodiments of this disclosure;

[0013] Figure 6 Examples of systems that can perform one or more processes described herein are shown according to some embodiments of this disclosure;

[0014] Figure 7 A flowchart is shown illustrating a method for processing audio data using a machine learning model that performs ASR processing and ITN processing, according to some embodiments of the present disclosure;

[0015] Figure 8 A flowchart is shown illustrating a method for processing audio data using a machine learning model that performs ASR processing and EOS and / or EOU processing, according to some embodiments of this disclosure;

[0016] Figure 9A This is a block diagram of an example generative language model system suitable for implementing some embodiments of the present disclosure;

[0017] Figure 9B This is a block diagram of an example generative language model including a converter encoder-decoder, suitable for implementing some embodiments of this disclosure;

[0018] Figure 9C This is a block diagram of an example generative language model, including a decoder-only converter architecture, suitable for implementing some embodiments of this disclosure;

[0019] Figure 10 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0020] Figure 11 This is a block diagram of an example data center suitable for implementing some embodiments of this disclosure. Detailed Implementation

[0021] This invention discloses a system and method for generating unified text for AI systems and applications using a speech recognition model. For example, the system can generate, acquire, receive, determine, and / or retrieve audio data representing speech from a user. As described herein, speech can be associated with utterances, such as utterances containing “We will rent you a GPU. The price is one hundred dollars” (and / or any other utterance). In some examples, the audio data can be preprocessed using one or more processing components (referred to as “audio processors”). For example, the audio processor can be configured to process the audio data to separate it into audio frames. The audio processor can then process the audio frames to generate features, such as Mel-spectral features, Fourier transform features, and / or any other type of audio feature. Furthermore, the audio processor can also generate embeddings and / or vectors representing the features of the audio frames.

[0022] In examples where an audio processor is used to preprocess audio data, input data representing features, embeddings, and / or vectors can be fed into a machine learning model. However, in other examples, the audio data can be directly fed into the machine learning model, which then processes the data using one or more of the processes described herein with respect to the audio processor. For example, the machine learning model may include one or more encoders configured to process the audio data and generate embeddings and / or vectors associated with audio frames of the audio data. In any of these examples, the machine learning model can be trained to perform various types of processing, such as ASR processing, PAC processing, EOS processing, EUO processing, and / or ITN processing, to generate uniform text containing punctuation, uppercase letters, EOS indicators, EUO indicators, and / or ITN formats.

[0023] For example, based at least on the processing of input data and / or audio data, a machine learning model can be trained to generate output data representing speech-related tags. In some examples, these tags can be associated with various types of processing that the machine learning model was trained on. For example, these tags can include: tags associated with ASR processing (ASR tags), which at least represent letters, parts of words, and / or words; tags associated with PAC processing (PAC tags), which represent punctuation marks (e.g., periods, commas, question marks, exclamation marks, etc.) and / or uppercase letters; tags associated with EOS processing (EOS tags), which represent symbols corresponding to the end of sentences; tags associated with EOU processing (EOU tags), which represent symbols corresponding to the end of utterances; and / or tags associated with ITN processing (ITN tags), which represent numbers, word-related symbols (e.g., $ for dollars, ° for degrees, etc.), and / or any other written word characters.

[0024] In some examples, the machine learning model can be trained to generate a tag set for each frame. For example, the machine learning model can generate a first tag set for the first frame, a second tag set for the second frame, a third tag set for the third frame, and so on. In some examples, the machine learning model can be trained to generate tag sets for groups of frames (e.g., two frames, five frames, ten frames, etc.). For example, the machine learning model can generate a first tag set for the first group of frames, a second tag set for the second group of frames, a third tag set for the third group of frames, and so on. In any example, the tag set can include one or more tags (e.g., each tag), and the machine learning model is trained to make predictions for these tags. For example, the tag set can include ASR tags, PAC tags, EOS tags, EUO tags, and / or ITN tags.

[0025] In some examples, the machine learning model can be further trained to generate output data representing the probabilities associated with a tag. For example, for a set of tags, the machine learning model can output corresponding probabilities indicating the likelihood that a given tag is associated with a frame and / or a set of frames. In some examples, the probabilities can be associated with a range, such as 0 to 1, 0 to 100, and / or any other range of values.

[0026] In some examples, one or more processing components (and / or one or more decoders of a machine learning model) can then be configured to use the output data to generate unified text associated with the speech; this can be referred to as a “tag processor.” As described herein, the unified text may contain punctuation marks, uppercase letters, EOS symbols, EOU symbols, and / or ITN formatted text (e.g., normalized format, also known as “normalized text”). To generate the unified text, the tag processor can be configured to select one or more tags from each tag set using probabilities, and then use the selected tags to generate the unified text. In some examples, to select tags, the tag processor may use a first technique (e.g., greedy decoding) that involves selecting the corresponding tag associated with the highest probability from each tag set. The tag processor can then use the selected tags to generate the unified text. For example, the unified text may contain letters, word parts, words, punctuation marks, uppercase letters, EOS symbols, EOU symbols, numbers, word symbols, etc., associated with the selected tags.

[0027] Additionally or alternatively, in some examples, to select tokens, the token processor may use a second technique (e.g., beam search / flash decoding) to select a set of tokens from each token set that are associated with several highest probabilities. In these examples, the token processor may then use the set of tokens selected from the token set to determine the uniform text. For example, the token processor may use one or more language models trained to output uniform text to process the set of tokens and the letters, word parts, words, punctuation marks, uppercase letters, EOS symbols, EOU symbols, numbers, and / or word symbols associated with those tokens. While these examples describe the token processor as processing output data to generate uniform text, in other examples, the machine learning model may further use similar techniques to process the output data to generate uniform text (e.g., using one or more decoders).

[0028] While the examples above describe machine learning models trained to perform ASR, PAC, EUO, EOS, and / or ITN processing, in other examples, machine learning models may also be trained to perform one or more of these processing methods. For the first example, if the machine learning model is trained to perform ASR, EUO, and EOS processing, the output tags may include ASR tags, EUO tags, and EOS tags, and the uniform text may contain EUO and EOS symbols. For the second example, if the machine learning model is trained to perform ASR, PAC, and ITN processing, the output tags may include ASR tags, PAC tags, and ITN tags, and the uniform text may contain punctuation, uppercase letters, and be in a normalized format.

[0029] In some examples, the system may use one or more techniques to train a machine learning model to generate output data related to unified text. For example, the machine learning model may be trained to output ASR tokens, PAC tokens, EOS tokens, EOU tokens, and / or ITN tokens. To train the machine learning model, the system may generate, acquire, receive, determine, and / or retrieve training input data, such as audio data representing speech instances (e.g., utterances). Furthermore, the system may generate, acquire, receive, determine, and / or retrieve ground truth data related to the training input data. In some examples, the ground truth data may represent unified text instances corresponding to utterances. Additionally, or alternatively, in some examples, the ground truth data may represent instances of unified text in a tokenized form, such as including tokens that the machine learning model is training to make predictions.

[0030] The system can then use various techniques to train a machine learning model using training data (e.g., training input data and corresponding baseline fact data). For the first example, for instance when the baseline fact data represents tokens, the machine learning model can process the training input data and, at least based on that processing, generate output data representing tokens. One or more training engines can then determine one or more losses, at least based on comparing the output tokens with ground truth tokens, and use the losses to update one or more parameters and / or weights of the machine learning model. For the second example, for instance when the baseline fact data represents instances of unified text, the machine learning model can process the training input data and, at least based on that processing, generate unified text. The training engine can then determine one or more losses, at least based on comparing the output unified text with ground truth unified text, and use the losses to update one or more parameters of the machine learning model. While these are just some example techniques for training machine learning models, in other examples, the system can use the additional and / or alternative techniques described herein to train machine learning models.

[0031] As described herein, the system can use one or more techniques to generate training data. For example, in some instances, such as to generate sufficient training data for initial training of a machine learning model, the system can use one or more systems, models, modules, etc., to automatically generate baseline fact data. For instance, the system can use ASR systems, PAC systems, EOU / EOS systems, and / or ITN systems to generate uniform text containing punctuation, uppercase letters, EOU symbols, EOS symbols, and / or normalized formats. Furthermore, or alternatively, in some instances, such as to generate training data to test the accuracy of a machine learning model, the system can use user feedback to generate baseline fact data.

[0032] In some examples, a machine learning model may include any type of model, neural network (e.g., convolutional neural network, recurrent neural network, etc.), transformer, module, and / or processing component configured to perform one or more of the processes described herein. For example, a machine learning model may include any type of model comprising: an encoder configured to initially process audio data to generate audio embeddings, and a decoder configured to process the audio embeddings to generate output data associated with uniform text. In such examples, the machine learning model may include a feedback loop between the decoder and encoder to ensure that linguistic cues from portions of the text are fed back to the encoder.

[0033] In some examples, various types of techniques can be used to perform at least a portion of the processes described herein using machine learning models. For a first example, a system executing an additional application (e.g., a natural language application configured to interpret text (e.g., for one or more systems, such as a vehicle control system, robot control system, avatar communication system, etc.), a machine translation application configured to translate text from a first language to a second language, and / or a language model application configured to process text) can use machine learning models to generate unified text input into these applications. In such an example, since unified text can indicate the position of sentence and / or utterance ends related to speech, the system can use techniques that help improve processing performance to input unified text into the application. For example, the system can input certain portions of unified text into the application based on the position of sentence and / or utterance ends.

[0034] For the second example, a system providing an interactive application (such as a game application, a communication application, and / or a collaboration group application) can perform one or more processes using a machine learning model. For instance, the system can receive audio data representing speech from a user of the interactive application. The system can then process the audio using a machine learning model (e.g., by using one or more processes described herein) to generate unified text associated with the speech. Furthermore, the system can provide the unified text to one or more devices of one or more users of the interactive application, enabling these devices to output audio associated with the speech and display content associated with the unified text.

[0035] In some examples, machine learning models can be packaged as microservices—such as inference microservices (e.g., NVIDIA NIM)—which can contain containers (e.g., operating system (OS)-level virtualization packages) that may include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine." For example, an inference microservice can contain the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model is small enough (e.g., has a small enough number of parameters), the model can be contained within the container itself. In some embodiments, the machine learning models described herein can be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment and execution software, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations to provide low latency and high throughput for production applications such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning models described herein may be included as part of the microservice, along with acceleration infrastructure capable of deployment using a single command and / or orchestration and autoscaling using a container orchestration system on the acceleration infrastructure (e.g., reaching data center scale on a single device). Thus, an inference microservice may include machine learning models (e.g., models optimized for high-performance inference), inference runtime software for executing the machine learning models and providing inputs (e.g., user queries, prompts, etc.) and outputs / responses, and enterprise management software for providing health checks, identification, and other monitoring capabilities. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of the machine learning models. When replacing or updating, the software performing the replacement / update can maintain the user configurations of the inference runtime software and the enterprise management software.

[0036] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0037] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing one or more visual language models (VLMs), systems implementing one or more multimodal language models, systems using or deploying one or more inference microservices, systems combining one or more machine learning models deployed in services or microservices with OS-level virtualization packages (e.g., containers), systems combining one or more virtual machines (VMs), systems implementing one or more multimodal models, systems for performing synthetic data generation operations, systems implemented at least partially in data centers, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative 3D asset content creation, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0038] refer to Figure 1A , Figure 1AExamples of a process 100 for generating unified text using a machine learning model according to some embodiments of this disclosure are shown. It should be understood that such and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions performed by the entities described herein can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory.

[0039] Process 100 may include one or more audio processors 102 receiving audio data 104 representing speech. As described herein, the speech may be associated with a utterance, such as the utterance “We will rent you a GPU. The price is one hundred dollars.” Process 100 may then include audio processors 102 processing the audio data 104 to generate input data 106 for a machine learning model 108. For example, audio processors 102 may be configured to process audio data 104 to separate the audio data 104 into audio frames. As described herein, audio frames may contain arbitrary lengths, such as 25 milliseconds, 50 milliseconds, 75 milliseconds, and / or any other length portion of audio data 104. Audio processors 102 may then process the audio frames to generate features, such as Mel-spectral features, Fourier transform features, and / or any other type of feature representing speech. Furthermore, audio processors 102 may generate embeddings and / or vectors representing the features of the audio frames, wherein input data 106 may represent audio frames, features, embeddings, and / or vectors.

[0040] As described herein, the audio processor 102 may include any type of processing components configured to perform one or more of the processes described herein, such as one or more machine learning models, one or more neural networks, one or more encoders, one or more modules, one or more fast Fourier transform components, etc. Furthermore, although Figure 1A The example shows audio processor 102 as separate from machine learning model 108, but in other examples, machine learning model 108 may include audio processor 102. For example, as Figure 1B As shown in the example, the machine learning model 108 may include one or more encoders configured to perform at least a portion of the processing described herein with respect to the audio processor 102.

[0041] Then, process 100 may include a machine learning model 108 processing input data 106 (and / or, in some examples, audio data 104), and generating speech-related output data 110 based at least on that processing. In some examples, such as Figure 1A As shown in the example, output data 110 may represent at least labels 112(1)-112(4) (also referred to as “label 112” in the singular or “multiple labels 112” in the plural) and probabilities 114(1)-114(4) (also referred to as “probabilities 114” in the singular or “multiple probabilities 114” in the plural). In some examples, labels 112 may be associated with various types of processing performed during the training of machine learning model 108. For example, labels 112 may include at least ASR label 112(1) associated with ASR processing, PAC label 112(2) associated with PAC processing, EOS / EOU label 112(3) associated with EOS / EOU processing, and / or ITN label 112(4) associated with ITN processing.

[0042] In some examples, the markers 112 associated with different types of processing can represent specific types of text, enabling the machine learning model 108 to generate the unified text described herein. For example, the ASR marker 112(1) can represent at least letters, characters, sentences (or parts thereof), parts of words (sub-words), and / or words, enabling the machine learning model 108 to perform ASR processing. Furthermore, the PAC marker 112(2) can represent at least punctuation marks (e.g., periods, commas, question marks, exclamation marks, etc.) and / or uppercase letters, enabling the machine learning model 108 to perform PAC processing. Additionally, the EOS / EOU marker 112(3) can represent at least one or more symbols associated with the end of a sentence and / or one or more symbols associated with the end of a utterance, enabling the machine learning model 108 to perform EOS / EOU processing. Furthermore, the ITN marker 112(4) can represent at least numbers, word-related symbols (e.g., $ for dollars, ° for degrees, etc.), and / or any other written characters, enabling the machine learning model 108 to perform ITN processing.

[0043] In some examples, the machine learning model 108 can be trained to generate a tag set for each frame. For example, the machine learning model 108 can generate a first tag set 112 for the first frame, a second tag set 112 for the second frame, a third tag set 112 for the third frame, and so on. In some examples, the machine learning model 108 can be trained to generate multiple tag sets 112 for groups of frames (e.g., two frames, five frames, ten frames, etc.). For example, the machine learning model 108 can generate a first tag set 112 for the first group of frames, a second tag set 112 for the second group of frames, a third tag set 112 for the third group of frames, and so on. In any example, a tag set may include one or more tags 112 (e.g., each tag 112), and the machine learning model 108 is trained to predict these tags. For example, a tag set 112 may include an ASR tag 112 (1), a PAC tag 112 (2), an EOS / EOU tag 112 (3), and / or an ITN tag 112 (4).

[0044] For example, Figure 2 Examples of generating a set of tags for audio data frames according to some embodiments of the present disclosure are shown. As shown, machine learning model 108 can generate output data (e.g., output data 110) that includes at least a first tag 202(1) associated with one or more first frames 204(1) of audio data (e.g., audio data 104), a second tag 202(2) associated with one or more second frames 204(2) of audio data, a third tag 202(3) associated with one or more third frames 204(3) of audio data, and so on, up to a final tag 202(N) associated with one or more final frames 204(N) of audio data. As described herein, in some examples, one or more sets of tags 202(1)-(N) (e.g., each set) may include ASR tag 112(1), PAC tag 112(2), EOS / EOU tag 112(3), and / or ITN tag 112(N).

[0045] review Figure 1A For example, probability 114 may indicate the likelihood that tag 112 is associated with a frame and / or a group of frames. In some examples, machine learning model 108 may be trained to generate a corresponding probability 114 associated with each tag 112. However, in other examples, machine learning model 108 may be trained to generate a set of corresponding probabilities for each tag 112. Furthermore, in some examples, probability 114 may be associated with a range, such as 0 to 1, 0 to 100, and / or any other range of values.

[0046] like Figure 1AAs further illustrated in the examples, process 100 may include one or more tag processors 116 for processing output data 110 and, at least based on this processing, generating text data 118 representing uniform text associated with speech. For example, tag processor 116 may be configured to select one or more tags 112 from each tag set 112 using probability 114, and then generate uniform text using the selected tags 112. In some examples, to select tags 112, tag processor 116 may use a first technique (e.g., greedy decoding) that includes selecting the corresponding tag 112 associated with the highest probability 114 from each tag set 112. Furthermore, or alternatively, in some examples, to select tags 112, tag processor 116 may use a second technique (e.g., cluster search / flash decoding) that includes selecting a set of tags associated with multiple highest probabilities 114 from each tag set 112.

[0047] For example, Figure 3 Examples of techniques for selecting tags associated with frames and / or groups of frames, according to some embodiments of this disclosure, are shown. As shown, tags 302(1)-302(O) (also referred to in the singular as “tag 302” or in the plural as “multiple tags 302”) can be associated with probabilities 304(1)-304(O) (also referred to in the singular as “probability 304” or in the plural as “multiple probabilities 304”). Figure 3 In the example, tags 302 can be arranged starting from the highest probability 304 associated with the first tag 302(1) and in descending order of probability 304 until the last tag 302(0). Therefore, the first technique 306 (e.g., a greedy decoding technique) could include selecting the first tag 302(1) associated with the highest probability 304(1). Furthermore, the second technique 308 (e.g., a cluster search / flash decoding) could include selecting a group of tags 302(1)-302(5) associated with the five highest probabilities 304(1)-304(5). Although Figure 3 The example description selects the group to contain five tags 302(1)-302(5) associated with the five highest probabilities 304(1)-304(5), but in other examples, the group may contain any number of tags 302 associated with any number of highest probabilities 304.

[0048] Return to reference Figure 1AIn the example, the tag processor 316 can then use the selected tags 112 to generate uniform text. For the first example, and for the first technique used to select tags 112, the tag processor 116 can generate uniform text to include letters, word parts, words, punctuation marks, uppercase letters, EOS symbols, EOU symbols, numbers, word-related symbols, etc., associated with the selected tags 112. For the second example, and for the second technique, the tag processor 116 can use one or more language models 120 to process the tag group 112, and the letters, word parts, words, punctuation marks, uppercase letters, EOS symbols, EOU symbols, numbers, word-related symbols, etc., associated with the tags 112. At least based on this processing, the language model 120 can be configured to select a sequence of tags 112 to generate the most meaningful uniform text (e.g., containing the correct language).

[0049] As described in this article, by executing Figure 1A In process 100, the unified text represented by text data 118 can be associated with ASR processing, PAC processing, EOU / EOS processing, and / or ITN processing. For the first example, if machine learning model 108 is trained to perform ASR processing and EOU / EOS processing, the labels 112 output by machine learning model 108 may include ASR label 112(1) and EOU / EOS label 112(3), and the unified text may include EOU symbols and EOS symbols. For the second example, if machine learning model 108 is trained to perform ASR processing, PAC processing, and ITN processing, the labels 112 output by machine learning model 108 may include ASR label 112(1), PAC label 112(2), and ITN label 112(4), and the unified text may contain punctuation marks, uppercase letters, and be in a normalized format.

[0050] For example, Figures 4A to 4B Examples of different types of processing that a machine learning model 108 can perform according to some embodiments of the present disclosure are shown. As illustrated, in... Figures 4A to 4BIn each example, machine learning model 108 can process audio data 402, which includes the utterance “We will rent you a GPU. The price is one hundred dollars.” Therefore, for the first example, machine learning model 108 may be trained only to perform ASR processing. Thus, machine learning model 108 can be used to generate text data 404(1) representing text containing “We will rent you a GPU for one hundred dollars.” (referred to as “ASR text”). Next, for the second example, machine learning model 108 can be trained to perform both ASR processing and ITN processing simultaneously. Therefore, machine learning model 108 can be used to generate first text data 406(1) representing first unified text containing “We will rent you a GPU for $100.” In this second example, the first unified text contains “$100” instead of “one hundred dollars” in the ASR text of the first example.

[0051] Next, for the third example, machine learning model 108 can be trained to perform both ASR and PAC processing simultaneously. Therefore, machine learning model 108 can be used to generate second text data 406(2), which represents a second unified text containing "We will rent you a GPU. The price is one hundred dollars". In the third example, the second unified text contains correct punctuation and capitalization compared to the ASR text in the first example. Next, for the fourth example, machine learning model 108 can be trained to perform both ASR and EOS / EOU processing. Therefore, machine learning model 108 can be used to generate third text data 406(3), which represents a third unified text containing "We will rent you a GPU / The price is one hundred dollars #". In this fourth example, the third unified text contains both the " / " symbol for indicating EOS and the "#" symbol for indicating EOS, compared to the ASR text in the first example. However, in other examples, the unified text may contain any other symbols to indicate EOS and / or EOU.

[0052] Next, for the fifth example, machine learning model 108 can be trained to perform ASR processing, PAC processing, and ITN processing. Thus, machine learning model 108 can be used to generate fourth text data 406(4), which represents fourth unified text containing “We will rent you a GPU. The cost is $100”. In this fifth example, compared to the ASR text in the first example, the fourth unified text contains correct punctuation, correct capitalization, and “$100”. Next, for the sixth example, machine learning model 108 can be trained to perform ASR processing, PAC processing, and EOU / EOS processing. Thus, machine learning model 108 can be used to generate fifth text data 406(5), which represents fifth unified text containing “We will rent you a GPU. / The cost is one hundred dollars. #”. In this sixth example, compared to the ASR text in the first example, the fifth unified text contains correct punctuation, current capitalization, a “ / ” symbol for indicating EOS, and a “#” symbol for indicating EUS.

[0053] Next, for the seventh example, machine learning model 108 can be trained to perform ASR processing, EUO / EOS processing, and ITN processing. Therefore, machine learning model 108 can be used to generate sixth text data 406(6), which represents the sixth unified text containing “We will rent you a GPU. / The cost is $100. #”. In this seventh example, compared to the ASR text in the first example, the sixth unified text contains the “ / ” symbol to indicate EOS, the “#” symbol to indicate EUO, and “$100”. Finally, for the eighth example, machine learning model 108 can be trained to perform ASR processing, PAC processing, EUO / EOS processing, and ITN processing. Therefore, machine learning model 108 can be used to generate seventh text data 406(7), which represents the seventh unified text containing “We will rent you a GPU. / The cost is $100. #”. In this eighth example, compared to the ASR text in the first example, the seventh unified text contains the correct punctuation, the correct capitalization, the “ / ” symbol to indicate EOS, the “#” symbol to indicate EUO, and “$100”.

[0054] Figure 1BAn example of a machine learning model 122 (which may be similar to and / or represent machine learning model 108) according to some embodiments of the present disclosure is shown. This model can be trained to generate unified text. As shown, audio data 104 can be directly input into machine learning model 122. Machine learning model 122 may include encoder 124 configured to process audio data 104 and generate embeddings 126 associated with frames of audio data 104, such as similar to audio processor 102. Furthermore, machine learning model 122 may also include decoder 128 configured to process embeddings 126 and generate speech-associated output data 130 based at least on this processing. As described herein, decoder 128 may include, but is not limited to, recurrent neural network decoders, connectionist temporal classification decoders, transformer decoders, and / or any other type of decoder. Furthermore, output data 130 may represent tokens similar to output data 110, and / or output data 130 may represent unified text, such as similar to text data 118.

[0055] As described in this paper, the machine learning model 108 can be trained to perform various types of processing, such as ASR processing, PAC processing, EOS / EOU processing, and / or ITN processing. Therefore, Figure 5A A data flow diagram illustrating the process for training a machine learning model 108 to perform various types of processing, according to some embodiments of the present disclosure, is shown. As shown, the machine learning model 108 can be trained using training data 502. In some examples, training data 502 may include speech instances (e.g., utterances) similar to audio data 104, and / or vectors and / or embeddings associated with the speech instances, similar to input data 106. Training data 502 may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data (e.g., audio data representing user speech)), and / or a combination thereof.

[0056] The machine learning model 108 can be trained using training data 502 and corresponding baseline fact data 504. As shown, in some examples, the baseline fact data 504 may include unified text 506 corresponding to the speech instances associated with the training data 502. Additionally, or alternatively, in some examples, the baseline fact data 504 may include tags 508 associated with the unified text 506, such as ASR tags, PAC tags, EOU / EOS tags, and / or ITN tags. As described herein, the baseline fact data 504 may be synthetically generated (e.g., generated by a computer model or rendered graph), realistically generated (e.g., designed and generated based on real-world data), machine-automated generated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated generated (e.g., by taggers or annotation experts), and / or a combination thereof. In some examples, for each instance of training data 502 (e.g., each speech instance, such as each utterance), there may be corresponding baseline fact data 504.

[0057] For example, in some examples, training data can be obtained by using audio recordings transcribed into uniform written text by human listeners and linguists, including punctuation, capitalization, sentence endings, utterance endings, and detextualization. Alternatively, in some examples, training data can be obtained using a system from a production-ready state-of-the-art model, which processes the audio data sequentially to predict text, then punctuation and capitalization, then sentence and utterance endings, followed by detextualization or any other processing order. However, in some examples, a hybrid approach of system cascading and human-computer interaction verification based on randomly selected training examples can be used to obtain training data to ensure accurate ground truth values.

[0058] For example, Figure 5B Examples of generating baseline fact data 504 for training machine learning model 108 according to some embodiments of the present disclosure are shown. As shown, the baseline fact data 504 can be generated using a series of processing components, such as an ASR component 510 configured to perform ASR processing, a PAC component 512 configured to perform PAC processing, an EOS / EOU component 514 configured to perform EOS / EOU processing, a language model 516, and / or an ITN component 518 configured to perform ITN processing.

[0059] For example, the ASR component 510 can first be used to process audio data 520 (which may represent and / or resemble training data 502) to generate text in spoken form, such as... Figure 4AThe first text data in the example is 404(1). This text can then be processed using the PAC component 512 to insert punctuation marks and correct the text to uppercase, for example, similar to... Figure 4A The second text data in the example is 406(2). Next, the text can be processed using the EOS / EOU component 514, which is configured to determine sentence endings and / or utterance endings associated with the text, such as those similar to... Figure 4B The fifth text data in the example is 406(5). After detecting the end of a sentence and / or the end of a utterance, the text can be processed using the language model 516, for example, to correct the text and / or improve the text structure if any errors exist in the ASR component 510. Finally, the text can be processed using the ITN component 518, which is configured to perform ITN processing, for example, by converting spoken words into written form, similar to... Figure 4B The seventh text data in the example is 406(7). The output from the processing may include text data 522 representing uniform text.

[0060] Although Figure 5B The example illustrates using each of ASR component 510, PAC component 512, EOS / EOU component 514, language model 516, and ITN processing 518 to process audio data 520. However, in other examples, one or more of these components may be used to process audio data 520. For instance, the processing performed on audio data 520 may be based on how the machine learning model 108 was trained. For the first example, if the machine learning model 108 is being trained to perform both ASR and ITN processing, then only ASR component 510 and ITN component 518 may be used to process audio data 520 in order to generate a language model that is compatible with ASR and ITN processing. Figure 4A The unified text represented by the first text data 106(1) in the example is similar to the unified text. For the second example, if the machine learning model 108 is being trained to perform ASR processing, PAC processing, and ITN processing, then only the ASR component 510, PAC component 512, and ITN component 518 can be used to process the audio data 520 in order to generate a unified text similar to... Figure 4B The fourth text data 106(4) in the example represents a unified text similar to the unified text.

[0061] review Figure 5AFor example, to train machine learning model 108, one or more training engines 524 may use one or more loss functions to measure the loss (e.g., error) of output 526 compared to baseline fact data 504. As described herein, in some examples, such as when baseline fact data 504 represents instances of uniform text 506, output 526 may also include uniform text instances determined by processing audio data 520 using machine learning model 108. Therefore, training engine 524 may compare instances of uniform text 506 with text instances from output 526 to measure loss. Alternatively, in some examples, such as when baseline fact data 504 represents tags 508, output 526 may also contain tags determined by the machine learning model processing audio data 520. Therefore, training engine 524 may compare tags 508 with tags from output 526 to measure loss.

[0062] For example, a loss function can be applied to the unified truth value (unified label). Specifically, a flag (e.g., a symbol) associated with PAC, EOS, EOU, and / or ITN can be inserted into the truth text to obtain unified text 506. Next, unified label 508 can be generated to represent the unified text 506 with the inserted flag, such that unified label 508 also represents PAC, EOS, EOU, and / or ITN. In this way, the loss function can be computed using the predicted label represented by the output data 526 and the unified label 508 represented by the baseline fact data 504. Any type of loss function, such as cross-entropy loss, mean squared loss, or any other type of loss, can then be used.

[0063] In these examples, any type of loss function can be used. For example, depending on the use case and training data, a difference function between the unified ground truth text and the predicted unified text can be used as the loss function. Furthermore, in some examples, there are no restrictions on the choice of loss function for the proposed method. Some available losses include: Connectionist Temporal Classification (CTC) loss, RNN-Transducer (RNN-T) loss, Cross-Entropy Loss, Sequence-to-Sequence Losses, Lattice-Free Maximum Mutual Information (LF-MMI), Minimum Word Error Rate (MWER) loss, Minimum Bayes Risk (MBR) loss, or a weighted combination of these losses. To build our prototype system, a hybrid loss can be used, which is a weighted combination of the CTC loss and the RNN-T loss computed based on the unified ground truth text and the predicted text.

[0064] In some examples, different outputs 526 may have different loss functions. For example, ASR tags 508 and / or unified text 506 related to ASR processing can use a first loss function, PAC tags 508 and / or unified text 506 related to PAC processing can use a second loss function, EOS / EOU tags 508 and / or unified text 506 related to EOS / EOU processing can use a third loss function, and / or ITN tags 508 and / or unified text 506 related to ITN processing can use a fourth loss function. In these examples, the loss functions can be combined to form a total loss (where one or more losses can be weighted), and the total loss can be used to train the machine learning model 108 (e.g., update the parameters of the machine learning model 108). In any example, backpropagation computation can be performed to recursively compute the gradient of the loss function with respect to the training parameters. In some examples, these gradients can be computed using the weights and / or biases of the machine learning model 108.

[0065] As described herein, the machine learning model 108 can be trained to perform one or more specific types of processing. For the first example, if the machine learning model 108 is trained to perform ASR processing and ITN processing, the baseline fact data 504 can be specific to ASR processing and ITN processing, such that the unified text 506 can at least be in ITN format, and / or the tokens 508 can contain ITN tokens. Therefore, the training engine 524 can determine a loss at least associated with ITN processing, for example, using a word error rate associated with ITN. For the second example, if the machine learning model 108 is being trained to perform ASR processing, PAC processing, and ITN processing, the baseline fact data can be specific to ASR processing, PAC processing, and ITN processing, such that the unified text can contain punctuation, use uppercase letters, and be in ITN format, and / or the tokens 508 can at least contain PAC tokens and ITN tokens. Therefore, the training engine 524 can use a first word error rate to determine a first loss associated with PAC processing and a second word error rate to determine a second loss associated with ITN processing.

[0066] In some examples, different machine learning models can be generated and / or trained for different languages, and / or a single machine learning model can be used for multiple languages. For example, monolingual models (one ASR model for each language region) or multilingual models (one ASR model for more than one language) can be generated for English and non-English languages. For example, the en-US ASR model would be a monolingual model focused on US English. The en-GB ASR model would be a monolingual model focused on British English, and so on. For another example, multilingual ASR models can be developed for popular languages ​​in Europe, the Middle East, and Africa (English, French, German, Dutch, Italian, Spanish, Portuguese, Arabic, Swahili, etc.). In other words, there may be no limit to the number of models supported by a multilingual model. For multilingual models, EOS / EOU tags may be needed to distinguish them from the superset of punctuation used in all the languages ​​served by the multilingual model.

[0067] In some examples, a machine learning model 108 can be generated and / or trained to perform code-switching. A code-switching scenario refers to a situation where a user speaks in their first language, then switches to a second language, and then returns to their first language or continues using the second language. In some cases, there is no limit to the number of different languages ​​used in a code-switching utterance. Furthermore, in some cases, there is no limit to the number of times a language is switched or changed within a utterance. For example, a user might speak in English for 2 minutes, then switch to German for 2 minutes, and then switch back to English for 1 minute. In such examples, a unified ASR model for code-switching scenarios can output unified text corresponding to the spoken language and the text, where the language switching is automatic. Furthermore, the unified ASR model can learn this during training, where it is trained / taught to recognize unified text from audio using the code-switching audio and the corresponding ground truth text.

[0068] In some examples, a set of punctuation marks can be selected to represent the output, depending on how the developer wants to train the machine learning model 108 and / or on the different languages ​​that the machine learning model 108 is trained on. For example, the developer can choose some specific ways to represent punctuation marks. For example, the developer can choose to use two spaces for a comma, three spaces for a period, four spaces for a question mark, or a dash for an exclamation mark. Thus, by performing one or more of the processes described herein, the machine learning model 108 can be trained to generate text containing a set of punctuation marks. For example, the baseline fact data 504 used to train the machine learning model 108 can contain the set of punctuation marks required by the developer in a uniform text so that the machine learning model 108 learns that set of punctuation marks during training.

[0069] Figure 6 An example of a system 602, capable of performing one or more processes described herein, according to certain embodiments of this disclosure, is illustrated. As shown, system 602 (which may represent and / or include example computing device 1000 and / or example data center 1100) may include one or more processors 604 (which may resemble and / or include one or more central processing units 1006 and / or one or more graphics processing units 1008) and memory 606 (which may resemble and / or include memory 1004). For example, memory 606 may store audio processor 102, machine learning model 108, and tagging processor 116. Furthermore, processor 604 may execute audio processor 102, machine learning model 108, and / or tagging processor 116 to perform one or more processes described herein.

[0070] In addition, such as Figure 6As illustrated in the example, system 602 can receive audio data 104 and / or send text data 118 to one or more client devices 608 (which may also be similar to and / or include example computing device 1000). For example, client device 608 may use one or more input devices (e.g., one or more microphones) to generate audio data 104. After generating audio data 104, client device 608 may send audio data 104 to system 602 for processing.

[0071] Furthermore, client device 608 can perform one or more processes using text data 118. For the first example, after receiving text data 118, client device 608 can use one or more output devices 610 to provide unified text, such as a display that presents unified text. For the second example, after receiving text data 118, client device 608 can use one or more additional processing components 612 configured to further process the unified text. For example, processing component 612 can use one or more natural language understanding applications, machine translation applications, and / or any other type of processing application to process the unified text. Although... Figure 6 The example shows that client device 608 includes processing component 612, but in other examples, system 602 may also include processing component 612.

[0072] In some examples, the machine learning model 108 can be packaged as a microservice (e.g., an inference microservice (e.g., NVIDIA NIM)), which can contain containers (e.g., operating system (OS) level virtualization packages) that may contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine". For example, the inference microservice may include the container itself and the model (e.g., weights and biases). In some cases, such as when the machine learning model 108 is small enough (e.g., has a sufficiently small number of parameters), the model may be contained within the container itself. In some embodiments, the machine learning model 108 described herein can be deployed as an inference microservice to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment execution software, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations to provide low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The machine learning model 108 described herein may be included as part of the microservice, as well as acceleration infrastructure capable of deployment using a single command and / or orchestration and autoscaling using a container orchestration system on the acceleration infrastructure (e.g., reaching data center scale on a single device). Thus, an inference microservice may include a machine learning model (e.g., a model optimized for high-performance inference), inference runtime software for executing the machine learning model 108 and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identification, and other monitoring. In some embodiments, the inference microservice may include software for performing in-situ replacements and / or updates to the machine learning model 108. During replacement or update, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0073] Now for reference Figure 6 and Figure 7Each block of methods 600 and 700 described herein contains a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Methods 600 and 700 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 600 and 700 can be provided by standalone applications, services, or managed services (standalone or in combination with other managed services) or plug-ins to another product, to name a few. Furthermore, methods 600 and 700 are combined with... Figure 1A These methods 600 and 700 are described by way of example. However, these methods 600 and 700 may be performed additionally or alternatively by any system or combination of systems, including but not limited to the systems described herein.

[0074] Figure 7 A flowchart of a method 700 for processing audio data using a machine learning model performing ASR processing and ITN processing, according to some embodiments of the present disclosure, is shown. Method 700 may include, at block B702, generating an embedding associated with audio data representing speech. For example, audio processor 102 may process audio data 104 and, at least based on this processing, generate input data 106 representing the embedding. As described herein, the embedding may be associated with frames of the audio data. Furthermore, although... Figure 1A The example shows audio processor 102 as separate from machine learning model 108, but in other examples, audio processor 102 may include a portion of machine learning model 108 (e.g., one or more encoders of machine learning model 108).

[0075] At block B704, method 700 may include: processing the embedding-related input data based at least on a machine learning model to generate output data representing one or more first tokens associated with automatic speech recognition and one or more second tokens associated with inverse text normalization. For example, machine learning model 108 may process input data 106 and generate output data 110 representing ASR token 112(1) and ITN token 112(4) based at least on the processing. As described herein, in some examples, output data 110 may also represent PAC token 112(2) and / or EOS / EOU token 112(3). Furthermore, in some examples, output data 110 may represent one or more probabilities 114(1) associated with ASR token 112(1) and / or one or more probabilities 114(4) associated with ITN token 112(4).

[0076] Method 700 at block B706 may include: generating normalized text associated with speech based on at least one or more first tags and one or more second tags. For example, tag processor 116 may use ASR tag 112 and ITN tag 112(4) to generate normalized text (e.g., unified text), such as text in written form. Furthermore, in some examples, tag processor 116 may use PAC tag 112(2) to generate normalized text containing one or more punctuation marks and / or uppercase letters, and / or use EOS / EOU tag 112(3) to generate text containing one or more EOS symbols and / or one or more EUO symbols. Although Figure 1A The example shows the tag processor 116 as separate from the machine learning model 108, but in other examples, the tag processor 116 may include a portion of the machine learning model 108 (e.g., one or more decoders of the machine learning model 108).

[0077] Method 700 at block B708 ​​may include performing one or more operations using normalized text. For example, one or more operations may include: rendering the normalized text using one or more client devices, further processing the normalized text using one or more additional processing units (e.g., one or more applications), and / or performing any other type of operation.

[0078] Figure 8 A flowchart of a method 800 for processing audio data using a machine learning model that performs ASR processing and EOS and / or EOU processing, according to some embodiments of the present disclosure, is shown. Method 800 may include, at block B802, generating output data based at least on processing speech-associated input data using the machine learning model, the output data representing one or more first tokens associated with automatic speech recognition and one or more second tokens associated with at least one of one or more sentence-ending symbols or one or more utterance-ending symbols. For example, machine learning model 108 may process input data 106 (and / or audio data 104) and, at least based on this processing, generate output data 110 representing ASR token 112 (1) and EOS / EOU token 112 (3). As described herein, in some examples, output data 110 may also represent PAC token 112 (2) and / or ITN token 112 (4). In addition, in some examples, output data 110 may represent one or more probabilities 114(1) associated with ASR tag 112(1) and / or one or more probabilities 114(3) associated with EOS / EOU tag 112(3).

[0079] Method 800 at block B804 may include: generating speech-related text based on at least one or more first tags and one or more second tags, the text containing at least one of one or more sentence-ending symbols or one or more utterance-ending symbols. For example, tag processor 116 may use ASR tag 112 and EOS / EOU tag 112(3) to generate text (e.g., uniform text) containing EOS symbols and EOU symbols. Furthermore, in some examples, tag processor 116 may use PAC tag 112(2) to generate text containing one or more punctuation marks and / or uppercase letters, and / or use ITN tag 112(4) to generate text to include normalized formatting.

[0080] Method 800 at block B806 may include performing one or more operations using text. For example, one or more operations may include rendering text using one or more client devices, further processing text using one or more additional processing units (e.g., one or more applications), and / or performing any other type of operation.

[0081] Example language model

[0082] In at least some embodiments, language models such as Large Language Models (LLMs), Visual Language Models (VLMs), Multimodal Language Models (MMLMs), and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered “large” because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. LLMs / VLMs / MMLMs / etc. can be implemented for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / etc. in a user-specified style, tone, and / or format. In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein may be specifically designed for text processing, while in others, a multimodal LLM may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format) and / or video. For example, a Visual Language Model (VLM) or more specifically a Multimodal Language Model (MMLM) may be implemented to accept images, video, audio, text, 3D designs (e.g., CAD) and / or other input data types and / or generate or output images, video, audio, text, 3D designs and / or other output data types.

[0083] Various types of LLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / VLM / MMLM / etc.

[0084] In various embodiments, unsupervised learning can be used to train LLM / VLM / MMLM / etc., where LLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0085] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0086] In some embodiments, an LLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / action / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0087] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0088] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in the curatorial dataset, for example, but not limited to this.

[0089] Figure 9A This is a block diagram of an example generative language model system 900 applicable to implementing at least some embodiments of this disclosure. Figure 9A In the example shown, the generative language model system 900 includes a retrieval augmentation (RAG) component 992, an input processor 905, a tokenizer 910, an embedding component 920, a plugin / API 995, and a generative language model (LM) 930 (which may include LLM, VLM, multimodal LM, etc.).

[0090] At a high level, the input processor 905 can receive input 901, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 930 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 901 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In some implementations where the generative LM 930 is capable of handling multimodal input, input 901 may include text (or may omit text) along with image data, audio data, video data, design data, USD data, etc. This can be a combination of input data and / or other types of input data (such as, but not limited to, the data described herein). Taking raw input text as an example, the input processor 905 can prepare the raw input text in various ways. For example, the input processor 905 can perform various types of text filtering to remove noise from the relevant text content (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.). In an example involving stop words (common words that often have little semantic meaning), the input processor 905 can remove stop words to reduce noise and allow the generative LM 930 to focus on more meaningful content. The input processor 905 can apply text normalization, for example, by converting all characters to lowercase, removing accent marks, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency. These are just a few examples; other types of input processing can be applied.

[0091] In some embodiments, RAG component 992 (which may include one or more RAG models, and / or may be performed using generative LM 930 itself) may be used to retrieve additional information to be used as part of input 901 or a prompt. RAGs can be used to enhance input to LLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, such as where specific knowledge is required. RAG component 992 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0092] For example, in some embodiments, in addition to the data retrieved using RAG component 992, input 901 may also be generated using query or model input (e.g., questions, requests, etc.). In some embodiments, input processor 905 may analyze input 901 and communicate with RAG component 992 (or in embodiments, RAG component 992 may be part of input processor 905) to identify relevant text and / or other data to provide to generative LM 930 as additional context or information source, typically from which to identify responses, answers, or outputs 990. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 992 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits the chatbot related to a specific product sale or service, the RAG component 992 can retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history, along with the current inquiry / request, as part of the generative LM 930 as input 901.

[0093] RAG component 992 can use various RAG techniques. For example, it can use naive RAG ( The document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of the RAG component 992, and the embeddings of the chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to the generative LM 930 to generate output.

[0094] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0095] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0096] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM / etc. Instead of providing the model with data chunks extracted from larger documents (which may result in a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with data chunks extracted from larger documents), Graph RAG can also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0097] In any embodiment, the RAG component 992 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or the embedded model.

[0098] The tokenizer 910 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 930 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 910 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0099] Embedding component 920 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 920 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0100] In some implementations where input 901 includes image data / video data, etc., input processor 901 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 920 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 901 includes audio data, input processor 901 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 920 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 901 includes video data, input processor 901 may extract frames or apply resizing to extracted frames, and embedding component 920 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 901 includes multimodal data, the embedded component 920 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0101] Other components of the generative LM 930 and / or generative LM system 900 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 920 can apply the encoded representation of the input 901 to the generative LM 930, and the generative LM 930 can process the encoded representation of the input 901 to generate an output 990, which may include response text and / or other types of data.

[0102] As described herein, in some embodiments, the generative LM 930 may be configured to access or use (or be able to access or use) plugins / APIs 995 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 930 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 992) to access one or more plugins / APIs 995 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 995 to the plugin / API 995, which can process the information and return an answer to the generative LM 930, which can then use the response to generate output 990. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 995 until an output 990 that resolves each query / question / request / process / action / etc. from input 901 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 992, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 995).

[0103] Figure 9B This is a block diagram of an example implementation scheme, where the generative LM 930 includes a converter encoder-decoder. For example, suppose the input text (e.g., “Who discovered gravity”) is tokenized (e.g., by...) Figure 9A The tokenizer 910) is used for tokens such as words, and each token is encoded (e.g., by...). Figure 9A The embedding component 920 is a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Thus, (e.g., the resulting) embeddings can be applied to one or more encoders 935 of the generative LM 930.

[0104] In the example implementation, encoder 935 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In the example converter architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a sequence of vectors, pass each vector through the self-attention layer, then through the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for a token pair can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector encoding the input. Attention projection layer 940 can transform the context vector into attention vectors (keys and values) for decoder 945.

[0105] In the example implementation, decoder 945 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence, and a feedforward network. Similar to encoder 935, in the example converter architecture, each token (e.g., a word) flows through a separate path in decoder 945. During the first pass, decoder 945, classifier 950, and generation mechanism 955 can generate a first token, and generation mechanism 955 can apply the generated token as input during a second pass. This process can be repeated cyclically, generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes applying token embeddings of positionally encoded composite sequences as input to decoder 945, generating one token at a time (called autoregression) until a symbol or token indicating the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on earlier positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In the example implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention operation in encoder 935, except that it creates its queries from the layers below it and obtains keys and values ​​(e.g., matrices) from the output of encoder 935.

[0106] Therefore, decoder 945 can output some decoded (e.g., vector) representation of the input applied during a particular pass. Classifier 950 can include a multi-class classifier comprising one or more neural network layers and a softmax operation that transforms logit probabilities into probabilities, the neural network layers projecting the decoded (e.g., vector) representation onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary). Thus, generation mechanism 955 can select or sample words or tokens based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby generating each word or token sequentially. Generation mechanism 955 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token representing the end of the response is selected or sampled, at which point generation mechanism 955 can output the generated response.

[0107] Figure 9C This is a block diagram of an example implementation where the generative LM 930 includes a decoder-only converter architecture. For example, Figure 9C The decoder 960 can be used with Figure 9B The decoder 945 operates similarly, except... Figure 9C Each decoder 960 omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, decoders 960 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to decoder 960. Figure 9B Similar to decoder 945, each tag (e.g., a word) can flow through a separate path in decoder 960, and decoder 960, classifier 965, and generation mechanism 970 can use autoregression to generate one tag at a time sequentially until a symbol or tag indicating the end of the response is predicted. Classifier 965 and generation mechanism 970 can be combined with... Figure 9B The classifier 950 and the generation mechanism 955 operate similarly, wherein the generation mechanism 970 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label representing the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0108] Example computing device

[0109] Figure 10This is a block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. The computing device 1000 may include an interconnect system 1002 directly or indirectly coupled to the following devices: a memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., one or more displays), and one or more logic units 1020. In at least one embodiment, one or more computing devices 1000 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1008 may include one or more vGPUs, one or more CPUs 1006 may include one or more vCPUs, and / or one or more logic units 1020 may include one or more virtual logic units. Accordingly, one or more computing devices 1000 may include discrete components (e.g., a full GPU dedicated to computing device 1000), virtual components (e.g., a portion of the GPU dedicated to computing device 1000), or a combination thereof.

[0110] although Figure 10 The various boxes are shown as being connected to lines via interconnect system 1002, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1018 (such as a display device) may be considered I / O component 1014 (e.g., if the display is a touchscreen). As another example, CPU 1006 and / or GPU 1008 may include memory (e.g., memory 1004 may represent a storage device in addition to the memory of GPU 1008, CPU 1006, and / or other components). In other words, Figure 10 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 10 Within the scope of computing devices.

[0111] Interconnect system 1002 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1002 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 1006 may be directly connected to memory 1004. Further, CPU 1006 may be directly connected to GPU 1008. In cases where there is a direct connection or point-to-point connection between components, interconnect system 1002 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 1000.

[0112] The memory 1004 may include any of a variety of computer-readable media. The computer-readable media may be any available medium accessible by the computing device 1000. The computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0113] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1004 may store computer-readable instructions (e.g., representing programs and / or program elements, such as operating systems). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1000. As used herein, computer storage media does not include the signal itself.

[0114] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0115] CPU 1006 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. Each CPU 1006 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. CPU 1006 may include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1000, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 1000 may also include one or more CPUs 1006.

[0116] In addition to or replacing CPU 1006, one or more GPUs 1008 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 may be integrated GPUs (e.g., having one or more CPUs 1006) and / or one or more GPUs 1008 may be discrete GPUs. In embodiments, one or more GPUs 1008 may be a coprocessor of one or more CPUs 1006. GPUs 1008 may be used by computing device 1000 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1008 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1008 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPUs 1008 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 1006 via a host interface). GPU 1008 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1004. GPU 1008 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1008 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0117] In addition to or replacing CPU 1006 and / or GPU 1008, one or more logic units 1020 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1006, one or more GPUs 1008, and / or one or more logic units 1020 may execute any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 1020 may be part of and / or integrated into one or more CPUs 1006 and / or GPUs 1008, and / or one or more logic units 1020 may be discrete components or otherwise external to CPUs 1006 and / or GPUs 1008. In embodiments, one or more logic units 1020 may be coprocessors of one or more CPUs 1006 and / or one or more GPUs 1008.

[0118] Examples of logic unit 1020 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0119] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers that allow the computing device 1000 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 1010 may include components and functions that allow communication via any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or wireless bandwidth), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet). In one or more embodiments, the logic unit 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) for directly transmitting data received via the network and / or via the interconnect system 1002 to one or more GPUs 1008 (e.g., the memory of one or more GPUs 1008).

[0120] I / O port 1012 allows computing device 1000 to be logically coupled to other devices including I / O component 1014, presentation component 1018, and / or other components, some of which may be built into (e.g., integrated into) computing device 1000. Illustrative I / O component 1014 includes microphone, mouse, keyboard, joystick, gamepad, game controller, disc satellite dish, scanner, printer, wireless device, etc. I / O component 1014 provides a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some instances, the input may be passed to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1000 (as described in more detail below). The computing device 1000 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 1000 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 1000 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0121] The power supply 1016 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 can provide power to the computing device 1000 so that the components of the computing device 1000 can operate.

[0122] One or more presentation components 1018 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 1018 may receive data from other components (e.g., GPU 1008, CPU 1006, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0123] Example Data Center

[0124] Figure 11 An example data center 1100 that can be used in at least one embodiment of this disclosure is shown. The data center 1100 may include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0125] like Figure 11 As shown, the data center infrastructure layer 1110 may include a resource coordinator 1112, grouped computing resources 1114, and node computing resources (“nodes CR”) 1116(1)-1116(N), where “N” represents any integer, a positive integer. In at least one embodiment, the nodes CR 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW”) devices, and network network interfaces (“NW”). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more nodes CR1116(1)-1116(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1116(1)-1116(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR1116(1)-1116(N) may correspond to virtual machines (VMs).

[0126] In at least one embodiment, the grouped computing resources 1114 may include individual groups of nodes CR1116 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The individual groups of nodes CR1116 within the grouped computing resources 1114 may include the group's computing, networking, memory, or storage resources, which may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR1116, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0127] Resource coordinator 1112 may be configured or otherwise control one or more nodes CR1116(1)-1116(N) and / or grouped computing resources 1114. In at least one embodiment, resource coordinator 1112 may include a Software Design Infrastructure (“SDI”) management entity for data center 1100. Resource coordinator 1112 may include hardware, software, or some combination thereof.

[0128] In at least one embodiment, such as Figure 11 As shown, framework layer 1120 may include a job scheduler 1128, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. Framework layer 1120 may include a framework for software 1132 supporting software layer 1130 and / or one or more applications 1142 supporting application layer 1140. Software 1132 or application 1142 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1120 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 1138 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM(Hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 1128 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of data center 1100. Configuration manager 1134 may be able to configure different layers, such as software layer 1130 and framework layer 1120 including Spark and distributed file system 1138 for supporting large-scale data processing. Resource manager 1136 may be able to manage cluster or group computing resources mapped to or allocated for supporting distributed file system 1138 and job scheduler 1128. In at least one embodiment, cluster or group computing resources may include group computing resources 1114 at data center infrastructure layer 1110. Resource manager 1136 may coordinate with resource coordinator 1112 to manage these mapped or allocated computing resources.

[0129] In at least one embodiment, the software 1132 included in software layer 1130 may include software used by at least a plurality of portions of nodes CR1116(1)-1116(N), grouped computing resources 1114, and / or the distributed file system 1138 of framework layer 1120. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0130] In at least one embodiment, the application 1142 included in the application layer 1140 may include one or more types of applications used by at least a plurality of portions of nodes CR1116(1)-1116(N), grouped computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments).

[0131] In at least one embodiment, any of the configuration manager 1114, resource manager 1136, and resource coordinator 1112 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can protect the data center operator of data center 1100 from making potentially poor configuration decisions and may prevent underutilized and / or poorly performing portions of the data center.

[0132] According to one or more embodiments described herein, data center 1100 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 1100. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1100 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0133] In at least one embodiment, the data center 1100 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to train or execute information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0134] Example network environment

[0135] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 10 The implementation is carried out on one or more instances of computing device 1000, for example, each device may include similar components, features and / or functions of computing device 1000. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 1100, an example of which is relative to the data center 1100 described herein. Figure 11 To describe in more detail.

[0136] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. A network can include multiple networks or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0137] A compatible network environment may include one or more peer-to-peer network environments—in which case the network environment may not include a server—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0138] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for software supporting the software layer and / or one or more applications supporting the application layer. The software or application may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, free and open-source software web application frameworks, such as those used for large-scale data processing (e.g., "big data") using distributed file systems.

[0139] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions may be distributed across multiple locations from a central or core server (e.g., across one or more data centers distributed across states, regions, countries, globally, etc.). If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0140] One or more client devices may include the information described in this article. Figure 10At least some of the components, features, and functions of one or more example computing devices 1000 described. By way of example and not limitation, the client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0141] This disclosure can be described in the general context of computer code or machine-usable instructions (including computer-executable instructions, such as program modules) that are executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network.

[0142] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as meaning only one element, or a combination of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0143] This document provides a detailed description of the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have anticipated that the claimed subject matter may also be embodied in other ways in combination with other current or future techniques to include combinations of different steps or steps similar to those described in this document. Furthermore, although the terms “step” and / or “box” may be used herein to refer to different elements of the method employed, such terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless and only if the order of individual steps is explicitly described.

[0144] Example paragraph

[0145] A: A method comprising: generating an audio embedding associated with audio data representing user speech; processing input data associated with the audio embedding based at least on a machine learning model to generate output data representing tags associated with the user speech, at least a first portion of the tags being associated with automatic speech recognition and at least a second portion of the tags being associated with inverse text normalization; generating normalized text representing the user speech based at least on the tags; and performing one or more operations using the normalized text.

[0146] B: As described in paragraph A, wherein: at least a third part of the mark is associated with at least one of the sentence end or the utterance end; and the normalized text includes at least one of the first indication of the sentence end or the second indication of the utterance end.

[0147] C: The method as described in paragraph A or paragraph B, wherein: at least a third portion of the mark is associated with at least one of one or more punctuation marks or one or more uppercase letters; and the normalized text includes the one or more punctuation marks or the one or more uppercase letters.

[0148] D: The method as described in any one of paragraphs A, C, and D, wherein: the output data further represents a probability associated with the marker; and the generation of the normalized text representing the user's speech is also based at least on the probability.

[0149] E: The method as described in any one of paragraphs A, D, and E, wherein: at least the first portion of the marker associated with the automatic speech recognition includes at least one of the following: one or more first markers representing one or more letters; one or more second markers representing one or more parts of one or more first words; or one or more third markers representing one or more second words; and at least the second portion of the marker associated with the inverse text normalization includes at least one of the following: one or more fourth markers representing one or more digits; or one or more fifth markers representing one or more symbols associated with one or more third words.

[0150] F: A method as described in any of paragraphs AE, wherein the tag is associated with one or more first frames of the audio data, and wherein the method further comprises: processing the input data at least based on the machine learning model to generate second output data representing a second tag associated with the user's speech, at least a first portion of the second tag being associated with the automatic speech recognition, and at least a second portion of the second tag being associated with the inverse text normalization, wherein generating the normalized text representing the user's speech is also based at least on the second tag.

[0151] G: The method as described in any one of paragraphs AF, wherein performing the one or more operations using the normalized text comprises at least one of: causing at least a portion of the normalized text to be processed using one or more second machine learning models; or causing output associated with at least a portion of the normalized text to be presented.

[0152] H: The method as described in any one of paragraphs A, B, and C further includes: acquiring second audio data representing second user speech and baseline fact data representing one or more second tags associated with the second user speech, at least a portion of the one or more second tags being associated with the inverse text normalization; generating one or more second audio embeddings associated with the second audio data; processing second input data associated with the one or more second audio embeddings based at least on the machine learning model to generate second output data representing one or more third tags associated with the second user speech; and updating one or more parameters associated with the machine learning model based at least on the one or more third tags and the one or more second tags.

[0153] 1: A system comprising: one or more processors configured to: process input data associated with user speech based at least on a machine learning model; generate output data representing: one or more first tokens representing at least one or more letters; and one or more second tokens representing one or more digits or one or more symbols, said one or more digits or one or more symbols representing one or more words; generate text representing user speech and containing said one or more digits or said one or more symbols based at least on said one or more first tokens and said one or more second tokens; and perform one or more operations using said text.

[0154] J: The system as described in paragraph I, wherein the one or more processors are further configured to generate, at least based on audio data representing the user's speech, one or more embedded input data corresponding to one or more frames associated with the audio data.

[0155] K: The system as described in paragraph I or paragraph J, wherein: the one or more second tokens are associated with inverse text normalization; and the text includes normalized text corresponding to the user's speech.

[0156] L: The system as described in any one of paragraphs IK, wherein: the output data further represents one or more third markers associated with at least one of the sentence ending or the utterance ending; and the text representing the user's speech is also generated based on at least the one or more third markers, and includes at least one of a first indication representing the sentence ending or a second indication representing the utterance ending.

[0157] M: The system as described in any one of paragraphs IL, wherein: the output data further represents one or more third markers associated with one or more punctuation marks; and the text representing the user's speech is generated at least based on the one or more third markers and includes the one or more punctuation marks.

[0158] N: The system as described in any one of paragraphs IM, wherein: the output data further represents one or more first probabilities associated with the one or more first markers and one or more second probabilities associated with the one or more second markers; and the text representing the user's speech is generated at least based on the one or more first probabilities and the one or more second probabilities.

[0159] O: The system as described in any one of paragraphs IN, wherein: the output data is associated with one or more first frames of audio data representing the user's speech; the one or more processors are further configured to process the input data at least based on the machine learning model to generate second output data associated with one or more second frames of the audio data, the second output data representing one or more third tokens, the one or more third tokens representing at least one of the one or more numbers or the one or more symbols, the one or more numbers or the one or more symbols representing the one or more words; and the text is also generated at least based on the one or more third tokens.

[0160] P: The system as described in any one of paragraphs 10, wherein performing the one or more operations using the text includes at least one of: causing at least a portion of the text to be processed using one or more second machine learning models; or resulting in output associated with at least a portion of the text.

[0161] Q: The system as described in any one of paragraphs IP, wherein the machine learning model comprises at least: one or more encoders for processing the audio data to generate the input data; and one or more decoders for processing the input data to generate the output data representing the one or more first tags and the one or more second tags.

[0162] R: A system as described in any of the paragraphs IQ, wherein the system includes at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models. Systems that perform operations using one or more multimodal language models (VLMs); systems that perform operations using one or more conversational AI operations; systems that generate synthetic data; systems that render at least one of virtual reality content, augmented reality content, or mixed reality content; systems that implement one or more multimodal language models; systems that use or deploy one or more inference microservices; systems that include the deployment of one or more machine learning models and operating system-level virtualization packages (e.g., containers) in services or microservices; systems that contain one or more virtual machines (VMs); systems that are at least partially implemented in a data center; or systems that are at least partially implemented using cloud computing resources.

[0163] S: One or more processors, comprising: processing circuitry for generating normalized text related to user speech based on at least one or more tokens, wherein the one or more tokens are generated based on at least the following operations: an encoder associated with a machine learning model processes audio data representing the user speech to generate a first output; and a decoder associated with the machine learning model processes the first output to generate a second output representing the one or more tokens.

[0164] T: One or more processors as described in paragraph S, wherein the machine learning model is deployed as an inference microservice, the inference microservice including the machine learning model and an operating system (OS) level virtualization package, the OS level virtualization package including software for executing the machine learning model and enterprise management software for performing one or more telemetry operations in association with the machine learning model.

Claims

1. A method comprising: Generate audio embeddings associated with audio data representing user speech; The system processes input data associated with the audio embedding based at least on a machine learning model to generate output data representing tags associated with the user's speech, wherein at least a first portion of the tags is associated with automatic speech recognition and at least a second portion of the tags is associated with inverse text normalization. Based at least on the markers, generate normalized text representing the user's speech; as well as Perform one or more operations using the normalized text.

2. The method according to claim 1, wherein: At least the third part of the mark is associated with at least one of the sentence ending or the end of a discourse; and The normalized text includes at least one of a first indication at the end of the sentence or a second indication at the end of the discourse.

3. The method according to claim 1, wherein: At least a third portion of the mark is associated with at least one of one or more punctuation marks or one or more uppercase letters; and The normalized text includes at least one of the one or more punctuation marks or the one or more uppercase letters.

4. The method according to claim 1, wherein: The output data also represents the probability associated with the label; and The generation of the normalized text representing the user's speech is also based at least on the probability.

5. The method according to claim 1, wherein: The first portion of the marker associated with the automatic speech recognition includes at least one of the following: One or more first markers representing one or more letters; One or more second markers representing one or more parts of one or more first words; or One or more third markers representing one or more second words; and The second portion of the tag related to the inverse text normalization includes at least one of the following: One or more fourth markers representing one or more digits; or One or more fifth marks representing one or more symbols associated with one or more third words.

6. The method according to claim 1, wherein, The marker is associated with one or more first frames of the audio data, and the method further includes: The input data is processed at least based on the machine learning model to generate second output data representing a second tag associated with the user's speech, wherein at least a first portion of the second tag is associated with the automatic speech recognition, and at least a second portion of the second tag is associated with the inverse text normalization. The generation of the normalized text representing the user's speech is also based at least on the second tag.

7. The method according to claim 1, wherein, Performing one or more operations using the normalized text includes at least one of the following: This enables the use of one or more second machine learning models to process at least a portion of the normalized text; or This results in an output that is associated with at least a portion of the normalized text.

8. The method according to claim 1, further comprising: Acquire second audio data representing a second user's speech and baseline fact data representing one or more second tags associated with the second user's speech, at least a portion of the one or more second tags being associated with the inverse text normalization; Generate one or more second audio embeddings associated with the second audio data; The machine learning model processes the second input data associated with the one or more second audio embeddings to generate second output data representing one or more third tags associated with the second user's speech. as well as Update one or more parameters associated with the machine learning model based at least on one or more third tags and one or more second tags.

9. A system comprising: One or more processors are used for: At least based on a machine learning model, the input data associated with the user's speech is processed to generate output data representing the following: One or more first markers representing at least one or more letters; as well as One or more second markers representing one or more numbers or one or more symbols, wherein the one or more numbers or one or more symbols represent one or more words; Based at least on the one or more first markers and the one or more second markers, generate text representing the user's speech and containing the one or more numbers or the one or more symbols; as well as Use the text to perform one or more operations.

10. The system according to claim 9, wherein, The one or more processors are further configured to generate, at least based on audio data representing the user's speech, one or more embedded input data corresponding to one or more frames associated with the audio data.

11. The system according to claim 9, wherein: The one or more second tags are associated with inverse text normalization; and The text includes normalized text corresponding to the user's voice.

12. The system according to claim 9, wherein: The output data also represents one or more third markers associated with at least one of the sentence ending or discourse ending; and The text representing the user's speech is also generated based on at least one or more third markers and includes at least one of a first indication indicating the end of the sentence or a second indication indicating the end of the utterance.

13. The system according to claim 9, wherein: The output data also represents one or more third markers associated with one or more punctuation marks; and The text representing the user's speech is also generated based on at least one or more third markers and includes one or more punctuation marks.

14. The system according to claim 9, wherein: The output data also represents one or more first probabilities associated with the one or more first tags and one or more second probabilities associated with the one or more second tags; and The text representing the user's speech is also generated based on at least one or more first probabilities and one or more second probabilities.

15. The system according to claim 9, wherein: The output data is associated with one or more first frames of audio data representing the user's speech; The one or more processors are further configured to process the input data based at least on the machine learning model, generating second output data associated with one or more second frames of the audio data, the second output data representing one or more third tokens, the one or more third tokens representing at least one of the one or more numbers or the one or more symbols, the one or more numbers or the one or more symbols representing the one or more words; and, The text is also generated based on at least one or more third tags.

16. The system according to claim 9, wherein, Performing one or more operations using the text includes at least one of the following: This enables the use of one or more second machine learning models to process at least a portion of the text; or This results in output associated with at least a portion of the text.

17. The system according to claim 9, wherein, The machine learning model includes at least: One or more encoders are used to process the audio data to generate the input data; and One or more decoders are used to process the input data to generate the output data representing the one or more first tags and the one or more second tags.

18. The system according to claim 9, wherein, The system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; Systems that include one or more machine learning models and operating system-level virtualization packages (e.g., containers) within a service or microservice; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. One or more processors, comprising: Processing circuitry for generating normalized text related to user speech based on at least one or more tokens, wherein the one or more tokens are generated based on at least the following operations: An encoder associated with a machine learning model processes audio data representing the user's speech to generate a first output; as well as The decoder associated with the machine learning model processes the first output to generate a second output representing the one or more tags.

20. One or more processors according to claim 19, wherein, The machine learning model is deployed as an inference microservice, which includes the machine learning model and an operating system (OS) virtualization package. The OS virtualization package includes software for executing the machine learning model and enterprise management software for performing one or more telemetry operations in association with the machine learning model.

Images