Multilingual automatic speech recognition

Abstract

A multilingual speech-to-text (STT) model of a multilingual automatic speech recognition (ASR) system is used and an audio sample is used as input to the multilingual STT model to determine a text transcription and one or more language indicators. The text transcription is associated with the audio sample and the one or more language indicators are each associated with a respective syntactic unit of one or more syntactic units of the text transcription. A monolingual language model (LM) of a plurality of monolingual LMs of the ASR system is identified using a language indicator of the one or more language indicators. The text transcription associated with the audio sample is optimized using the identified LM and using a subset of the text transcription as input to the identified LM.

Description

Technical Field

[0001] Various aspects and embodiments of this disclosure relate to automatic speech recognition, and more particularly to a multilingual automatic speech recognition pipeline using a multilingual speech-to-text model and language indicators. Background Technology

[0002] Automatic speech recognition typically involves acoustic speech-to-text machine learning models, which are trained to recognize a single language and transcribe streaming or recorded audio of that language into text. Language models and other post-processing operations, such as inverse text normalization, can be used to refine and enhance the text transcription generated by the speech-to-text models. Attached Figure Description

[0003] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0004] Figure 1A This is a block diagram of an example system architecture using a multilingual automatic speech recognition pipeline with a multilingual speech-to-text model and language indicators according to at least one embodiment;

[0005] Figure 1B This is a block diagram of an example architecture of a multilingual STT model according to at least one embodiment;

[0006] Figure 2 The illustration shows a multilingual STT training dataset generated from multiple monolingual STT training datasets according to at least one embodiment;

[0007] Figure 3 The illustration shows a multilingual STT training dataset generated from a code-switching training dataset according to at least one embodiment;

[0008] Figure 4 This is a flowchart of an example method for a multilingual automatic speech recognition pipeline using a multilingual speech-to-text model and language indicators according to at least one embodiment;

[0009] Figure 5 This is a block diagram of an example computing device according to at least one embodiment;

[0010] Figure 6 An example data center according to at least one embodiment is illustrated;

[0011] Figures 7A-7B The illustration depicts inference and / or training logic, according to at least one embodiment, used to perform inference and / or training operations;

[0012] Figure 8A It is a block diagram of an example generative language model system according to at least one embodiment;

[0013] Figure 8B It is a block diagram of an example implementation of a generative LM including a transformer encoder-decoder according to at least one embodiment;

[0014] Figure 8C It is a block diagram of an example implementation of a converter architecture in which the generative LM includes a decoder-only model according to at least one embodiment;

[0015] Figure 9 It is a block diagram of a computing system having two processing devices coupled to each other and multiple networks according to at least one embodiment;

[0016] Figure 10 It is a block diagram of a computing system having a CPU and a GPU in a single integrated circuit according to at least one embodiment; and

[0017] Figure 11 It is a block diagram of a computing system with a tensor core GPU according to at least one embodiment. Detailed Implementation

[0018] This disclosure relates to multilingual automatic speech recognition and transcription. Multilingual automatic speech recognition (ASR) typically involves acoustic speech-to-text machine learning models (STT models) trained to recognize a single language and transcribe streaming or recorded audio of that language into text. Therefore, the language being transcribed should be known beforehand to select an appropriate monolingual STT model. The text transcription generated by the STT model can be optimized and enhanced using language models (LMs) and other post-processing operations such as inverse text normalization (ITN, which can be rule-based or model-based). LMs and ITNs can also be language-specific.

[0019] Compared to monolingual ASR techniques such as those described above, multilingual ASR can offer several benefits. Multilingual ASR can use fewer models to recognize and transcribe speech in multiple languages, reducing the need to train and manage multiple STT models or LMs and minimizing the need for prior speech recognition. Multilingual ASR can also support code switching (e.g., alternating between two or more languages ​​within an audio sample). However, multilingual ASR faces additional challenges. Multilingual LMs and ITNs can be inefficient due to the size of these models. Furthermore, training multilingual LMs is complicated by the imbalance in the availability of training data across different languages ​​(e.g., English may have more available training data than some other languages). This can lead to poor performance in some languages, making mitigation using advanced data augmentation and balancing techniques both difficult and time-consuming. The architecture and training parameters of multilingual STT models and LMs may differ from those of equivalent monolingual models, which can further complicate the training process and prevent the reuse of existing training infrastructure.

[0020] This disclosure addresses these and other challenges by providing a multilingual ASR pipeline that includes a multilingual STT model and multiple monolingual LM models, as well as a monolingual ITN and other post-processing components. The multilingual STT model can be trained to recognize multiple languages ​​and transcribe them into text. The multilingual STT model can also be trained to annotate the transcribed text with relevant languages ​​using language indicators. Language indicators can be associated with individual grammatical units (such as paragraphs, sentences, words, morphemes, epigraphs, etc.) and placed in the transcription. For example, language markers... <en-us>It can be placed at the end of a transcribed sentence that is identified as US English.

[0021] In at least one embodiment, because language indicators can be part of the model's vocabulary, the multilingual STT model can use the same or similar model architecture, loss function, and other training components as the monolingual STT model. The multilingual STT model can be trained on a multilingual training dataset that includes relevant language indicators at appropriate locations (e.g., at the end of a sentence). The monolingual training dataset can be modified to include these indicators. Therefore, the benefit of the multilingual STT model is a smaller model and automatic language recognition while retaining the architectural and training benefits of the monolingual STT model.

[0022] In at least one embodiment, once the multilingual STT model has transcribed speech into text and inserted language indicators, the multilingual ASR system can use one or more language indicators to select an appropriate monolingual LM and other relevant monolingual post-processing operations (such as ITN). The multilingual ASR system can use the selected monolingual components to optimize and enhance the transcription. Thus, the multilingual ASR system can retain the benefits of a smaller post-processing model associated with a monolingual approach, as well as the benefits associated with training an LM separately due to imbalanced training data.

[0023] In at least one embodiment, the multilingual ASR system with a multilingual STT model and a monolingual post-processing component can also support code switching (when a speaker alternates between two or more languages ​​during a conversation). The multilingual STT model can be trained to group two (or more) language indicators with sentences or other grammatical units that include code switching. In the post-processing stage, an available bilingual LM can be used to handle code switching, or a combination of monolingual LMs can be used to handle code switching.

[0024] Figure 1A This is a block diagram of an example system architecture 100 using a multilingual speech-to-text model and language indicators according to at least one embodiment of an automatic multilingual speech recognition pipeline. System architecture 100 (also referred to herein as the "system") includes a network 110, client devices 120A-120N, data storage 130, and servers 140-170. In various embodiments, system 100 may include more than Figure 1A The components depicted may be more or less in different configurations. For example, system 100 may include additional servers, networks, etc. In another example, servers 140-170 may be combined.

[0025] Network 110 may include public networks (e.g., the Internet), private networks (e.g., LANs, WANs, VPNs, enterprise networks), wired networks (e.g., Ethernet), wireless networks (e.g., 802.11 Wi-Fi networks), cellular networks (e.g., 5G networks), routers, hubs, switches, server computers, or combinations thereof. In various embodiments, network 110 or its components may be associated with different organizations. For example, components of network 110 may be associated with Internet service providers (ISPs), mobile or cellular operators, cloud platform or Software as a Service (SaaS) providers, private or public enterprises, private homes or communities, etc. In at least one embodiment, network 110 (or its components) may be a physical or virtual interconnect within a single device, such as a PCIe bus, messaging system, or API.

[0026] Client devices 120A-120N can be personal computers (PCs), laptops, notebooks, mobile phones, smartphones, tablets, digital assistants, connected TVs (e.g., smart TVs), handheld gaming devices, game consoles, or any other computing device. Figure 5 The computer system may be an example of a client device. In various embodiments, client devices 120A-120N may also be referred to as "user devices". Client devices 120A-220N may run an operating system (OS) that manages the hardware and software of the client devices. Client devices 120A-120N may also include a web browser, application, or other software for interacting with servers 140-170. Client devices 120A-120N may be used by a user to initiate ASR processes (e.g., training and / or inference) on servers 140-170. Generally and as described herein, functions described in various embodiments as being performed by servers 140-170 may also, or alternatively, be performed on client devices 120A-120N in other embodiments. For example, in at least one embodiment, ASR inference may be performed on client devices 120A-120N. Additionally, functions belonging to specific components may be performed by different or multiple components operating together.

[0027] Data storage 130 can be an application for receiving, storing, and providing data. Data storage 130 can be a relational or non-relational database, a structured or unstructured database, a key-value store, a file system, or may conform to other data storage classifications. Data storage 130 can be made of various persistent or non-persistent storage devices (such as RAM, magnetic tape or drives, solid-state drives, optical disc drives, etc., e.g., see reference below). Figure 5 Other storage technologies discussed are supported. Data storage 130 may also include storage devices in a network topology, such as Storage Area Network (SAN), Network Attached Storage (NAS), cloud-provided storage, etc. Data storage 130 may be provided by one or more servers (not depicted). In at least one embodiment, data storage 130 is provided by server 140. Data storage 130 or its corresponding hardware may be centralized or distributed. Examples of database applications that may correspond to data storage 130 include MongoDB, MySQL, MariaDB, DynamoDB, PostgreSQL, etc. Data storage 130 may partition data into various storage units, buckets, tables, etc., based on the needs of one or more applications served by the data storage. In at least one embodiment, data storage 130 may store monolingual STT training datasets and / or multilingual STT training datasets (such as multilingual STT training dataset 132) used for training monolingual STT models and / or multilingual STT models.

[0028] Each of servers 140-170 can be a rack server, router computer, personal computer, portable digital assistant, mobile phone, laptop, tablet, netbook, desktop computer, virtual machine (VM), container, or any combination thereof. Figure 5 The computer system may be an example of a server. In various embodiments, each of servers 140-170 may be multiple computing devices, such as multiple rack servers in one or more data centers or multiple VMs in a cloud platform. In at least one embodiment, the functionality provided by servers 140-170 may alternatively be provided by a single server.

[0029] Server 140 includes an STT model training service 142, which can be used to perform various operations associated with training or fitting an STT model, such as data cleaning, data generation, data augmentation, regression, gradient calculation, backpropagation, loss calculation, etc. For example, STT model training service 142 can train a multilingual STT model 154 using a multilingual STT training dataset 132 stored in data storage 130. Server 150 includes an STT model inference service 152, which can be used to perform various operations associated with STT model inference, such as generative operations (e.g., sampling from a distribution), discriminative operations (e.g., classification), and other types of operations. For example, STT model inference service 152 can perform inference on the trained multilingual STT model 154. (Reference) Figures 7A-7B The example training and reasoning logic are also described.

[0030] The multilingual STT model 154 can be a speech-to-text machine learning model trained to recognize multiple languages ​​and transcribe these languages ​​into text. The multilingual STT model can also be trained to annotate the transcribed text with relevant languages ​​using language indicators. Language indicators can be part of a predefined vocabulary of the multilingual STT model 154 and can be distinct from any phoneme or glyph. The use and placement of language indicators can be learned from the structure of the multilingual STT training dataset 132, which will refer to... Figure 2-3 Further description is provided. In various embodiments, various model architectures can be used for the multilingual STT model 154. For example, the multilingual STT model 154 can be or may include a transformer (e.g., encoder-decoder, encoder-only, decoder-only), a recurrent neural network (e.g., LSTM), a convolutional neural network, a hidden Markov model, etc. In at least one embodiment, the architecture of the multilingual STT model 154 can be adapted to a monolingual STT model (e.g., without language indicators in the corresponding vocabulary) or to a multilingual STT model as described herein (e.g., with language indicators in the corresponding vocabulary). Therefore, reference data storage 130, server 140, and... Figures 7A-7B The described STT model training logic can be effectively used or reused to train both monolingual and multilingual STT models.154

[0031] Server 160 includes a language model training and / or inference service 162, which can be used to perform training or inference on one or more language models (such as language models 164A-164N). Although depicted as being performed by a single server / service, in at least one embodiment, training and inference can be partitioned among multiple servers / services. Similarly, different servers / services can be used to train different language models and / or perform inference on different language models.

[0032] Language models 164A-164N can be machine learning models trained to perform one or more language-related tasks on text output generated by an STT model or other components of the ASR pipeline. For example, a language model can be trained to correct low-probability transcriptions by correcting meaningless sequences of grammatical units (e.g., morphemes, words, etc.) into meaningful sequences. In another example, a language model can be trained to add or correct textual cues, such as capitalization and punctuation. In various embodiments, the language model can be monolingual, bilingual, or multilingual. The language model can be trained using a corresponding monolingual training dataset, bilingual training dataset, or multilingual training dataset. See also... Figures 8A-8C Examples of some types of language models that can be used are described.

[0033] Server 170 includes a reverse text normalization service 172 that can be used to perform reverse text normalization on transcripts using one or more of the ITN engines 174A - 174N. Although depicted as being performed by a single server / service, different servers / services can be used to perform ITN using different ITN engines.

[0034] The ITN engines among 174A - 174N can be machine learning models, rule sets, or other structures trained / constructed to perform one or more reverse text normalization tasks on the text output generated by the STT model, language model, or other components of the ASR pipeline. For example, an ITN engine can be trained / constructed to convert the text "one hundred and twenty - three" to "123", convert "doctor" to "Dr.", etc. In various embodiments, the ITN engine can be monolingual, bilingual, or multilingual.

[0035] Although the ASR pipeline of system 100 is depicted as having three components (multilingual speech - to - text, language model, and reverse text normalization), other embodiments can have more or fewer components than those depicted. For example, one embodiment can exclude reverse text normalization. In another example, an embodiment can include additional stages / components for removing background noise, isolating a speaker's voice from other voices, etc.

[0036] Figure 1B is a block diagram of an example architecture of a multilingual STT model 154 according to at least one embodiment. The multilingual STT model 154 can include an encoder layer 182 for receiving an acoustic speech signal 180 and generating an acoustic embedding. The encoder layer 182 can be, for example, a Conformer or FastConformer architecture. The acoustic embedding can be provided to a decoder layer 184 to predict multilingual text tokens 186A, multilingual punctuation tokens 186B, and / or language indicators 186C. The tokens 186A - C can be part of a multilingual vocabulary 188. Refer to Figure 2-3 for further description of the multilingual vocabulary and tokens. In at least one embodiment, a connectionist temporal classification (CTC) loss, an RNN transducer (RNNT) loss, or other loss functions can be used to train (e.g., by training service 142) the multilingual STT model 154.

[0037] Figure 2 The illustration shows a multilingual STT training dataset 220 generated from multiple monolingual STT training datasets 210A-210N according to at least one embodiment. The monolingual STT training datasets 210A-210N can be existing training datasets that can be used to train monolingual STT models. The monolingual STT training datasets 210A-210N can be combined with language indicators to generate the multilingual STT training dataset 220, which can be used to train multilingual STT models as described herein (e.g., the multilingual STT model 154 of Figure 1). In at least one embodiment, the multilingual STT training dataset 220 is the multilingual STT training set 132 of Figure 1.

[0038] Monolingual STT training datasets (such as Monolingual STT Training Dataset 210A) can be associated with a single language and / or dialect. For example, Monolingual STT Training Dataset 210A can be associated with the English language or the US English dialect. Monolingual STT Training Dataset 210A can include one or more training sample pairs, such as Monolingual Sample Pair 212A. Monolingual Sample Pair 212A includes audio samples and corresponding text transcriptions, such as the sentence "the quick brown fox jumps over the lazy dog". The grammatical units of the text transcription can be encoded using a monolingual vocabulary. For example, morphemes, words, etc., can be encoded as integers, vectors in an embedding space, etc. The vocabulary can include additional non-textual markers, such as the beginning of the sentence. <sos>and the end of the sentence <eos>Indicators. The vocabulary may include punctuation marks that can supplement or replace non-textual markers (e.g., comparison). Figure 2 of <sos>and <eos>Indicators and Figure 3 (Punctuation marks). The vocabularies of monolingual STT training datasets may differ. For example, monolingual STT training dataset 210A can use an English vocabulary, while monolingual STT training dataset 210N can use a German vocabulary. Different sample pairs within the training dataset may have the same vocabulary. For example, all sample pairs in training dataset 210A can use an English vocabulary.

[0039] A multilingual STT training dataset (such as multilingual STT training dataset 220) can be associated with multiple languages ​​and / or dialects. For example, multilingual STT training dataset 220 can be associated with English or German (and / or corresponding dialects). Multilingual STT training dataset 220 can include one or more training sample pairs, such as monolingual sample pairs 222A-222N. As described with reference to monolingual sample pair 212A, each training sample pair can include an audio sample and a corresponding text transcription. In contrast to monolingual STT training datasets 210A-N, multilingual STT training dataset 220 and associated sample pairs can be associated with a multilingual vocabulary, which can include morphemes, words, etc., from multiple languages ​​and / or dialects. The multilingual vocabulary can also include multilingual punctuation, uppercase and lowercase characters, and other language-specific features. Each sample pair can be associated with a subset of the multilingual vocabulary corresponding to a single language or dialect (such as English (sample pair 222A) or German (sample pair 222N)). The vocabulary may also include additional non-textual markers to indicate the language associated with the grammatical units. As depicted, such as English (US) <en-us>and German (German) <de-de>Such language indicators can be placed at the end of a sentence (e.g., in...). <eos>(Before the indicator). In various embodiments, the language indicator may be located at the beginning, end, and / or other positions relative to grammatical units (such as morphemes, words, sentences, paragraphs, etc.). In another embodiment, the language indicator may be located at the boundary between different languages.

[0040] To generate the multilingual STT training dataset 220, the monolingual STT training datasets 210A-210N can be augmented and combined. Augmenting the training dataset can include adding language indicators to sample pairs in each training dataset, as depicted in operations 202A-202B. Augmenting the training dataset can also include re-encoding sample transcriptions from their respective monolingual vocabularies into a language indicator-enhanced multilingual vocabulary for the multilingual training dataset. In various embodiments, other types of dataset augmentation can be used. Thus, augmented sample pairs can be combined to form a joint multilingual STT training dataset with a single multilingual vocabulary.

[0041] Figure 3 The illustration shows a multilingual STT training dataset 320 generated from a code-switching training dataset 310 according to at least one embodiment. The code-switching training dataset 310 includes one or more code-switching sample pairs, such as code-switching sample pair 312A. Reference Figure 2 The aspects described in the sample pairs can be similarly applied to code-switching sample pair 312A. For example, sample pair 312A can be associated with a vocabulary that includes multiple languages ​​involved in code-switching (e.g., English and German). The vocabulary may also include punctuation marks and / or non-textual markers (e.g., ...). Figure 3 It depicts a question mark, and Figure 2 Depicting <sos>and <eos>indicator).

[0042] To generate a multilingual STT training dataset 320 from the code-switched training dataset 310, multiple language indicators can be added to the sample pair 312A at operation 302 to form a multilingual sample pair 322A. Each of the multiple language indicators can indicate the language unit present in the code-switched syntax unit it refers to (e.g., in...). Figure 3 In the case of a sentence, the language is used. In various embodiments, such as references... Figure 2 As described, language indicators can be located in various positions with respect to the grammatical unit they refer to, such as at the beginning or end of a sentence, before or after punctuation marks or non-textual markers, etc.

[0043] Figure 4 This is a flowchart of an example method 400 using a multilingual speech-to-text model and language indicators according to at least one embodiment of an automatic multilingual speech recognition pipeline. Method 400 can be executed by processing logic that may include hardware (e.g., circuitry, dedicated logic, etc.), computer-readable instructions such as software or firmware (e.g., running on a general-purpose computing system or a dedicated machine), or a combination thereof. For example, the example system may include a memory and a processing device coupled to the memory device to perform operations including the block of method 400. Method 400 may also be associated with a set of instructions stored on a non-transitory computer-readable medium (e.g., a disk or optical disk, etc.). When executed by the processing device, the instructions may cause the processing device to perform operations including the block of method 400. In at least one embodiment, method 400 is... Figure 1A The method is performed by one or more servers 140-170 or client devices 120A-120N or components thereof. In at least one embodiment, method 400 is performed by... Figure 5 The computing system 500 executes the commands. In some embodiments, the commands may be executed simultaneously or in a different order than those described. Figure 4 The boxes depicted. Various embodiments may include Figure 4 Additional boxes not depicted in the text or Figure 4 A subset of the boxes depicted in the text.

[0044] At box 402, the processing logic trains a multilingual STT model for the multilingual ASR system using training data. This training data includes one or more training audio samples, one or more training text transcriptions each associated with a corresponding audio sample from the one or more training audio samples, and one or more training language indicators each associated with a corresponding text transcription from the one or more training text transcriptions. The multilingual STT model can be a multilingual STT model 154 of the ASR system 100, and can be trained by an STT model training service 142. The training data can be multilingual STT training datasets 132, 220, and / or 320. (See reference...) Figure 2 As described, the training data may include one or more sample pairs (e.g., sample pairs 222A-222N), each sample pair including a training audio sample and a training text transcription. The training transcription for each pair may include one or more languages ​​and / or dialects representing the presence of the audio-transcription pair (such as...). <en-us>or <de-de>One or more language indicators. A multilingual STT model can be associated with a multilingual vocabulary (e.g., multilingual vocabulary 188), which may include one or more language indicators. Reference Figures 7A-7B Example training and inference logic are also described. In at least one embodiment, the model architecture of the multilingual STT model corresponds to (e.g., is the same as) the model architecture of the monolingual STT model, and can be trained using the same training infrastructure (e.g., training service 142) as the monolingual STT model with multilingual training data.

[0045] At box 404, the processing logic uses the multilingual STT model of the multilingual ASR system and uses audio samples as input to the multilingual STT model to determine the text transcription associated with the audio samples and one or more language indicators, each associated with a corresponding grammatical unit in one or more grammatical units of the text transcription. The audio samples can be... Figure 1B The speech signal 180, and the text transcription may include a multilingual text token 186A and a multilingual punctuation token 186B. The language indicator may be a language indicator 186C. Processing logic can determine the text transcription and language indicator by performing inference on the multilingual STT model (such as by using the STT model inference service 152 and obtaining the text transcription and language indicator as the output of the multilingual STT model). In at least one embodiment, this determination can be based at least on the multilingual STT model processing the audio data corresponding to the audio sample.

[0046] In at least one embodiment, one or more grammatical units of text transcription are sentences, and one or more language indicators are each located after the punctuation mark or sentence end mark of the corresponding sentence. In other embodiments, grammatical units may be morphemes, words, clauses, paragraphs, etc., and language indicators may be located before and / or after the grammatical unit or its respective punctuation.

[0047] In at least one embodiment, two language indicators from one or more language indicators are each associated with a code-switching grammatical unit of one or more grammatical units of the text transcription. In at least one embodiment, the code-switching grammatical unit is a sentence, and the two language indicators are located after the sentence's punctuation marks (e.g., as shown in the image). Figure 3 (As depicted).

[0048] At box 406, the processing logic uses language indicators from one or more language indicators to identify a monolingual LM among multiple monolingual LMs in the ASR system. The multiple monolingual LMs can be LMs 164A-164N of Figure 1. The processing logic can identify a monolingual LM that has been trained to match the language represented by the language indicator. For example, the processing logic can be based on transcription... <en-us>Use the marker to select English LM.

[0049] At box 408, the processing logic removes one or more language indicators from the text transcription. After identifying one or more relevant LMs, the language indicators may no longer be used in the ASR pipeline. Furthermore, the language indicators may not be in the vocabulary of the identified language model. Therefore, the processing logic can remove language indicators from the text transcription before providing it to the LM or other post-processing components (e.g., the ITN engine). In at least one embodiment, the processing logic can also convert (e.g., re-encode) the text transcription from the multilingual vocabulary of the STT model to the monolingual vocabulary of the identified LM.

[0050] At box 410, the processing logic uses the identified LM and a subset of the text transcription as input to the identified LM to optimize the text transcription associated with the audio sample. The processing logic may enable other servers / services (e.g., LM training / inference service 162) to optimize the text transcription using the identified LM. The subset of text transcription may be text transcriptions without one or more language indicators (e.g., as removed at box 408). Optimizing the text transcription may include correcting syntactic or semantic errors (e.g., grammatical errors or meaningless phrases). In at least one embodiment, the processing logic optimizes the text transcription using other monolingual post-processing components (such as a monolingual ITN engine). In at least one embodiment, the processing logic determines the updated text transcription based at least a portion of the monolingual LM used to process the transcription and the associated language indicators of one or more language indicators.

[0051] In at least one embodiment, the processing logic uses the updated text transcription to perform one or more operations. The one or more operations may include: causing at least a portion of the updated text transcription to be presented using one or more display devices of the system; translating at least a portion of the updated text transcription into another language and causing it to be displayed using one or more display devices; or translating at least a portion of the updated text transcription into another language and causing audio output using synthesized sound corresponding to that portion of the updated text transcription.

[0052] In at least one embodiment, the processing logic processes audio segments based at least on a multilingual language model to generate an initial text transcription including one or more language indicators, and processes the initial text transcription and one or more language indicators using one or more monolingual language models to generate a final text transcription corresponding to the audio segments. In at least one embodiment, during the generation of the final text transcription, the processing logic removes one or more language indicators from the initial text transcription. In at least one embodiment, the processing logic trains a multilingual language model using training data including one or more second audio segments, one or more second text transcriptions each associated with a corresponding audio segment in one or more second audio segments, and one or more second language indicators each associated with a corresponding text transcription in one or more second text transcriptions. In at least one embodiment, the final text transcription is one of the following: stored on a device, visually rendered on a device, or used to generate synthesized audio output using a device. In at least one embodiment, the initial text transcription includes two different language indicators corresponding to code switching within the audio segment, and the final text transcription is generated using a first monolingual language model corresponding to a first language in the initial text transcription and a second monolingual language model corresponding to a second language in the initial text transcription. In at least one embodiment, the processing logic is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulated operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more Large Language Models (LLMs); a system for performing operations using one or more... A system that performs operations using multiple visual language models (VLMs); a system for performing operations using one or more multimodal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system that implements one or more multimodal language models; a system that uses or deploys one or more inference microservices; a system that includes one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers); a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0053] In some examples, one or more machine learning models described herein (e.g., STT models, language models, etc.) can be packaged as microservices (such as inference microservices (e.g., NVIDIA NIM)). These microservices can include containers (e.g., operating system (OS) level virtualization packages) that can include an application programming interface (API) layer, a server layer, a runtime layer, and / or a model "engine." For example, an inference microservice can include the container itself and one or more models (e.g., weights and biases). In some instances (such as when one or more machine learning models are small enough (e.g., have a sufficiently few parameters)), one or more models can be included within the container itself. In other instances (such as when one or more models are large), one or more models can be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container).

[0054] In these embodiments, one or more models can be accessed via one or more APIs, such as REST APIs. Thus, and in some embodiments, one or more machine learning models described herein can be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtimes and model optimizations for providing low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring).

[0055] The one or more machine learning models described herein can be included as part of a microservice, along with accelerated infrastructure, enabling deployment with a single command and / or orchestration and auto-scaling on the accelerated infrastructure using a container orchestration system (e.g., reaching data center scale on a single device). Thus, an inference microservice can include one or more machine learning models (e.g., those optimized for high-performance inference), inference runtime software for executing one or more machine learning models and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software providing health checks, identity, and / or other monitoring.

[0056] In some embodiments, the inference microserver may include software that performs in-situ replacement and / or updates to one or more machine learning models. When a replacement or update is performed, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0057] In some embodiments, the systems and methods described herein can be deployed in calling or smart kiosk applications. For example, a kiosk, tablet, smart display, or other device may include one or more onboard processors (e.g., CPU, GPU, deep learning accelerator, SoC) and memory and / or storage devices (e.g., for storing models, image databases, etc.). In some embodiments, the kiosk / tablet / display may (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located (e.g., in one or more data centers) servers / computing devices. In these examples, the kiosk may use one or more APIs (such as, but not limited to, REST APIs) to communicate with one or more machine learning models (e.g., STT models, language models, etc.) and / or image databases hosted on local and / or remote servers.

[0058] In one or more embodiments, the systems and methods described herein can be deployed in gaming applications. For example, game consoles, PCs, tablets, or other gaming devices may include one or more onboard and / or remote processors (e.g., CPUs, GPUs, deep learning accelerators, SoCs) and memory and / or storage devices (e.g., for storing game models, game assets, player data, etc.). These devices can use one or more machine learning models (e.g., STT models, language models, etc.) to enhance gameplay, generate real-time dynamic content, and personalize the user experience based on in-game behavior or pre-stored player profiles. In some embodiments, the system can be deployed in a cloud gaming environment (e.g., NVIDIA's GeForce Now). In this case, client devices (e.g., smart displays, tablets, or game controllers) can be used to interact with the game, while one or more machine learning models and / or visual rendering can occur on one or more remotely located server / computing devices (e.g., in one or more data centers). The language model, AI processing, and rendering described in this paper can operate in the cloud to process player input received from one or more end-user devices (e.g., based on controller, keyboard, mouse, joystick, AR / VR / MR / etc. input), generate appropriate in-game responses, render content, and send or transmit content to one or more end-user devices. During data reception and / or transmission with one or more end-user or edge devices, one or more data processing units (DPUs) and / or network interface cards (NICs) can be used.

[0059] In some embodiments, the systems and methods described herein can be deployed in video conferencing applications. For example, video conferencing devices (such as dedicated conferencing units, computers, tablets, and / or smartphones) may include one or more onboard processors (e.g., CPUs, GPUs, deep learning accelerators, SoCs) and memory and / or storage devices (e.g., for storing video, audio, or other communication-related data). The system can enhance video conferencing capabilities using one or more machine learning models (e.g., STT models, language models, etc.), including real-time or near real-time transcription, speaker diarization (speech segmentation), language translation, automatic speech recognition (ASR), and / or background noise reduction. In one or more embodiments, the system can enable users to interact with the video conferencing platform using natural language input. For example, users can issue voice commands to schedule, join, or leave a meeting or manage participants and screen sharing. During data reception and / or transmission with one or more end-users or edge devices, one or more data processing units (DPUs) and / or network interface cards (NICs) may be used.

[0060] In some embodiments, the systems and methods described herein can be deployed in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPUs)), direct memory access (DMA) systems and / or pixel processing engines (PPEs)), hardware-based optical flow accelerators (OFA), SoCs, etc.), and memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models). The robotic system can use these processors to execute one or more machine learning models (e.g., STT models, language models) that allow the robotic system to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects, or navigating the environment using sensors (such as cameras, LiDAR, RADAR, ultrasonic sensors, etc.). This system can use sensor fusion technology to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, RADAR, accelerometers) to create a comprehensive model of the robot's surrounding environment. This data can be processed locally on the robot or sent to a remote server for computationally intensive tasks such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g., sensor data, task status, or environmental conditions) can be uploaded to the cloud, where a centralized AI model can analyze optimized commands and distribute them across the entire cluster (fleet). In some embodiments, one or more machine learning models described herein (e.g., STT models, language models, etc.) can be used to allow the robot to perceive and reason about its environment and / or communicate with one or more other robots and / or humans in the environment. In some embodiments, the robot can (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) communicate with one or more locally hosted servers / computing devices and / or with one or more remotely located servers / computing devices (e.g., in one or more data centers).

[0061] In some embodiments, the systems and methods described herein can be deployed in in-vehicle infotainment (IVI) systems or in-cabin experience (IX) applications. For example, an infotainment system within a vehicle (e.g., a car, truck, drone, engineering equipment, robot, semi-autonomous vehicle, or autonomous vehicle) may include one or more onboard processors (e.g., CPU, GPU, hardware-based deep learning accelerator (DLA), hardware-based programmable vision accelerator (PVA) (which may include one or more vector processing units (VPUs)), direct memory access (DMA) systems and / or pixel processing engines (PPEs)), hardware-based optical flow accelerators (OFA), SoCs, etc.), memory and / or storage devices (e.g., for storing control algorithms, sensor data, and one or more machine learning models), and memory and / or storage devices (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to implement one or more machine learning models (e.g., STT models, language models) to achieve features such as voice control, personalized media recommendations, dynamic navigation, and real-time communication with other services via network connectivity. In-vehicle infotainment systems can also use natural language processing (NLP) models to enable voice-based interaction. One or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real-time or near real-time.

[0062] Figure 5 This is a block diagram of an example computing device 500 suitable for implementing some embodiments of the present disclosure. For example, computing device 500 may correspond to one or more Figure 1A The client devices 120A-120N and / or servers 140-170. The computing device 500 may include an interconnect system 502 directly or indirectly coupled to the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, input / output (I / O) ports 512, input / output components 514, a power supply 516, one or more presentation components 518 (e.g., one or more displays), and one or more logic units 520. In at least one embodiment, one or more computing devices 500 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 508 may include one or more vGPUs, one or more CPUs 506 may include one or more vCPUs, and / or one or more logic units 520 may include one or more virtual logic units. Accordingly, one or more computing devices 500 may include discrete components (e.g., a full GPU dedicated to computing device 500), virtual components (e.g., a portion of the GPU dedicated to computing device 500), or a combination thereof.

[0063] although Figure 5 The various blocks are shown as being connected to lines via interconnect system 502, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 518 (such as a display device) may be considered I / O component 514 (e.g., if the display is a touchscreen). As another example, CPU 506 and / or GPU 508 may include memory (e.g., memory 504 may represent a storage device in addition to the memory of GPU 508, CPU 506, and / or other components). In other words, Figure 5 The computing devices described are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are conceived in… Figure 5 Within the scope of computing devices.

[0064] Interconnect system 502 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 502 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 506 may be directly connected to memory 504. Further, CPU 506 may be directly connected to GPU 508. In cases where there is a direct connection or point-to-point connection between components, interconnect system 502 may include a PCIe link to perform that connection. In these examples, a PCI bus is not required in computing device 500.

[0065] The memory 504 may include any of a variety of computer-readable media. The computer-readable media may be any available medium accessible by the computing device 500. The computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable media may include computer storage media and communication media.

[0066] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 504 may store computer-readable instructions (e.g., representing programs and / or program elements, such as operating systems). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 500. As used herein, computer storage media does not include the signal itself.

[0067] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals (such as carrier waves or other transmission mechanisms) and include any information transmission medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the foregoing should also be included within the scope of computer-readable media.

[0068] CPU 506 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. Each CPU 506 may include one or more cores (e.g., 1, 2, 4, 8, 28, 72, etc.) capable of processing multiple software threads simultaneously. CPU 506 may include any type of processor and may include different types of processors depending on the type of computing device 500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 500, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplemental coprocessors such as math coprocessors, computing device 500 may also include one or more CPUs 506.

[0069] In addition to or replacing CPU 506, one or more GPUs 508 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. One or more GPUs 508 may be integrated GPUs (e.g., having one or more CPUs 506) and / or one or more GPUs 508 may be discrete GPUs. In embodiments, one or more GPUs 508 may be a coprocessor of one or more CPUs 506. GPUs 508 may be used by computing device 500 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 508 may be used for general-purpose computing on a GPU (GPGPU). GPUs 508 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPUs 508 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 506 via a host interface). GPU 508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 504. GPU 508 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 508 may generate pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0070] In addition to or replacing CPU 506 and / or GPU 508, one or more logic units 520 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 506, one or more GPUs 508, and / or one or more logic units 520 may perform any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 520 may be part of and / or integrated into one or more of CPUs 506 and / or GPUs 508, and / or one or more logic units 520 may be discrete components or otherwise external to CPUs 506 and / or GPUs 508. In embodiments, one or more logic units 520 may be coprocessors of one or more CPUs 506 and / or one or more GPUs 508.

[0071] Examples of logic unit 520 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0072] Communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable computing device 500 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 510 may include components and functions that allow communication over any of a plurality of different networks (e.g., wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wireless bandwidth), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.) and / or the Internet). In one or more embodiments, logic unit 520 and / or communication interface 510 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 502 to one or more GPUs 508 (e.g., the memory of one or more GPUs 508).

[0073] I / O port 512 enables computing device 500 to be logically coupled to other devices including I / O component 514, presentation component 518, and / or other components, some of which may be built into (e.g., integrated into) computing device 500. Illustrative I / O component 514 includes a microphone, mouse, keyboard, joystick, gamepad, game controller, disc satellite dish, scanner, printer, wireless device, etc. I / O component 514 provides a natural user interface (NUI) that handles air gestures, voice, or other physiological input generated by the user. In some instances, the input may be passed to appropriate network elements for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 500 (as described in more detail below). The computing device 500 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touchscreen technology, and combinations thereof, for attitude detection and recognition. Additionally, the computing device 500 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the computing device 500 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0074] Power supply 516 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 516 may provide power to computing device 500 so that the components of computing device 500 can operate.

[0075] One or more presentation components 518 may include displays (e.g., monitors, touchscreens, television screens, head-up displays (HUDs), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation component 518 may receive data from other components (e.g., GPU 508, CPU 506, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0076] Figure 6 An example data center 600 that can be used in at least one embodiment of this disclosure is shown. For example, data center 600 may include one or more Figure 1A Services 140-170. Data center 600 may include data center infrastructure layer 610, framework layer 620, software layer 630 and / or application layer 640.

[0077] like Figure 6 As shown, the data center infrastructure layer 610 may include a resource coordinator 612, grouped computing resources 614, and node computing resources ("nodes CR") 616(1)-616(N), where "N" represents any integer, a positive integer. In at least one embodiment, nodes CR 616(1)-616(N) may include, but are not limited to, any number of central processing units ("CPU") or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW"), and network network interface cards ("NW"). I / O devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more nodes CR616(1)-616(N) may correspond to a server having one or more of the aforementioned computing resources. Furthermore, in some embodiments, nodes CR616(1)-616(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CR616(1)-616(N) may correspond to virtual machines (VMs).

[0078] In at least one embodiment, the grouped computing resources 614 may include separate groups of nodes CR616 housed within one or more racks (not shown) or within a plurality of racks in data centers (also not shown) located in different geographical locations. The separate groups of nodes CR616 within the grouped computing resources 614 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of nodes CR616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0079] Resource coordinator 612 may be configured or otherwise controlled to control one or more nodes CR616(1)-616(N) and / or groups of computing resources 614. In at least one embodiment, resource coordinator 612 may include a Software Design Infrastructure ("SDI") management entity for data center 600. Resource coordinator 612 may include hardware, software, or some combination thereof.

[0080] In at least one embodiment, such as Figure 6 As shown, framework layer 620 may include a job scheduler 628, a configuration manager 634, a resource manager 636, and / or a distributed file system 638. Framework layer 620 may include a framework of software 632 supporting software layer 630 and / or one or more applications 642 of application layer 640. Software 632 or application 642 may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 620 may be, but is not limited to, a type of free and open-source software web application framework that can utilize the distributed file system 638 for large-scale data processing (e.g., "big data"), such as Apache Spark. TM (Hereinafter referred to as "Spark"). In at least one embodiment, job scheduler 628 may include Spark drivers to facilitate the scheduling of workloads supported by various layers of data center 600. Configuration manager 634 may be able to configure different layers, such as software layer 630 and framework layer 620 including Spark and distributed file system 638 for supporting large-scale data processing. Resource manager 636 may be able to manage cluster or group computing resources mapped to or allocated for supporting distributed file system 638 and job scheduler 628. In at least one embodiment, cluster or group computing resources may include group computing resources 614 at data center infrastructure layer 610. Resource manager 636 may coordinate with resource coordinator 612 to manage these mapped or allocated computing resources.

[0081] In at least one embodiment, the software 632 included in software layer 630 may include software used by at least a plurality of portions of nodes CR616(1)-616(N), grouped computing resources 614, and / or the distributed file system 638 of framework layer 620. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0082] In at least one embodiment, the application 642 included in the application layer 640 may include one or more types of applications used by at least a plurality of portions of nodes CR616(1)-616(N), grouped computing resources 614, and / or the distributed file system 638 of the framework layer 620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications (including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in combination with one or more embodiments).

[0083] In at least one embodiment, any of the configuration manager 634, resource manager 636, and resource coordinator 612 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free the data center operator of data center 600 from making potentially undesirable configuration decisions and potentially avoid underutilized and / or poorly performing portions of the data center.

[0084] According to one or more embodiments described herein, data center 600 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using the software and / or computing resources described above with respect to data center 600. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 600 by using weight parameters calculated through one or more training techniques (such as, but not limited to, those described herein).

[0085] In at least one embodiment, the data center 600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured to allow users to train or execute information inference services, such as image recognition, speech recognition, or other artificial intelligence services.

[0086] Figure 7A Inference and / or training logic 715 is illustrated for performing inference and / or training operations associated with one or more embodiments. For example, the inference and / or training logic can be used for... Figure 1A The multilingual STT models 154 and / or LM164A-164N are used for training and / or inference. The following combines... Figure 7A and / or Figure 7B Provide details about reasoning and / or training logic 715.

[0087] In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters using aspects of training and / or inference in one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0088] In at least one embodiment, any portion of the code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 may be a cache memory, dynamic random access memory ("DRAM"), static random access memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 701 is internal or external to the processor, for example, or including DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip versus off-chip storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0089] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of the neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, an arithmetic logic unit (ALU)). In at least one embodiment, code (such as graph code) causes weight or other parameter information to be loaded into the processor ALU based on the architecture of the neural network corresponding to the code. In at least one embodiment, any portion of the code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 705 is internal or external to the processor, for example, including DRAM, SRAM, flash memory, or some other type of storage, may depend on the available on-chip versus off-chip storage, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0090] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be the same storage structure. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0091] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units ("ALUs") 710 (including integer and / or floating-point units) for performing logical and / or mathematical calculations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activations stored in activation storage 720 are generated based on linear algebra and / or matrix-based mathematics performed by ALU 710 in response to execution instructions or other code, wherein weight values ​​stored in code and / or data storage 705 and / or code and / or data storage 701 are used as operands, and other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, may be stored in code and / or data storage 705 or code and / or data storage 701 or other on-chip or off-chip storage.

[0092] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 710 may be external to the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, the ALUs 710 may be included within an execution unit of a processor, or otherwise included in an ALU bank accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may be on the same processor or other hardware logic device or circuitry, while in another embodiment, they may be on different processors or other hardware logic devices or circuitries, or in some combination of the same and different processors or other hardware logic devices or circuitries. In at least one embodiment, any portion of activation storage 720 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0093] In at least one embodiment, the active memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 720 may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 720 is internal to or external to the processor, for example, or including DRAM, SRAM, flash memory, or certain other memory types, may depend on the available on-chip versus off-chip memory, the latency requirements for performing training and / or inference functions, the batch size of the data used in inference and / or training the neural network, or some combination of these factors. In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit ("ASIC"), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or other hardware such as field programmable gate array ("FPGA").

[0094] Figure 7B Inference and / or training logic 715 according to at least one or more embodiments is illustrated. In at least one or more embodiments, the inference and / or training logic 715 may include, but is not limited to, hardware logic, wherein computational resources, along with weight values ​​or other information corresponding to one or more layers of neurons within a neural network, are used dedicatedly or otherwise exclusively. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., data processing unit ("DPU") hardware, or field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown, each of code and / or data storage 701 and code and / or data storage 705 is associated with dedicated computing resources (e.g., computing hardware 702 and computing hardware 706), respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, with the results stored in active storage 720.

[0095] In at least one embodiment, each of the code and / or data storage 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of the neural network, such that an activation obtained from one storage / computation pair 701 / 702 of the code and / or data storage 701 and computing hardware 702 is provided as input to the next storage / computation pair 705 / 706 of the code and / or data storage 705 and computing hardware 706, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 701 / 702 and 705 / 706 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0096] In at least some embodiments, language models (such as Large Language Models (LLM), Visual Language Models (VLM), Multimodal Language Models (MMLM), and / or other types of generative artificial intelligence (AI)) can be implemented. These models are capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., in USD formats such as OpenUSD) based on context provided in input prompts or queries. In various embodiments, these language models can be considered "large" based on the model being trained on massive datasets and having an architecture with a large number of learnable network parameters (weights and biases) (e.g., millions or billions of parameters). LLMs / VLMs / MMLMs, etc., can be implemented to summarize textual data, analyze data, extract insights from data (e.g., text, images, videos, etc.), and generate new text / images / videos, etc., in a user-specified style, tone, and / or format. In various embodiments, the LLM / VLM / MMLM, etc., disclosed herein can be specifically used for text processing, while in other embodiments, a multimodal LLM can be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., in USD format) and / or video. For example, a visual language model (VLM) or a more general multimodal language model (MMLM) can be implemented to accept images, video, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, video and / or audio, text, 3D designs, and / or other output data types. In at least one embodiment, LM164A-164N of FIG1 are LLM, VLM, MMLM, etc.

[0097] In various embodiments, various types of LLM / VLM / MMLM architectures can be implemented. For example, different architectures can be implemented that use different techniques to understand and generate outputs such as text, audio, video, images, 2D and / or 3D design or asset data. In some embodiments, architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can be used for LLM / VLM / MMLM, while in other embodiments, transformer architectures (such as transformer architectures that rely on self-attention and / or cross-attention (e.g., between contextual data and textual data)) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / VLM / MMLM may also include one or more diffusion blocks (e.g., denoisers). The LLM / VLM / MMLM of this disclosure may include one or more encoder and / or decoder blocks. For example, for tasks involving language understanding, such as classification, sentiment analysis, question answering, and named entity recognition, discriminative models such as BERT (Bidirectional Encoder Representation from Transformer) or encoder-only models can be implemented. As another example, for tasks involving language and content generation, such as text completion, story generation, and dialogue generation, generative models such as GPT (Generative Pre-trained Transformer) or decoder-only models can be implemented. LLM / VLM / MMLM, etc., which include encoder and decoder components (such as T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting, and any architecture type (including, but not limited to, the architecture types described herein) can be implemented according to specific embodiments and one or more tasks performed using LLM / VLM / MMLM, etc.

[0098] In various embodiments, LLM / VLM / MMLM, etc., can be trained using unsupervised learning, whereby LLM / VLM / MMLM, etc., learn patterns from large amounts of unlabeled text / audio / video / image / design / USD, etc. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM, etc., extensively pre-trained on large amounts of unlabeled data, can be referred to as a base model and may excel at multiple tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM, etc., can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., tuning or adjusting cue or tagging to bias the language model towards a specific task or domain), and / or using other fine-tuning or customization techniques to optimize the model for a specific task and / or use within a specific domain.

[0099] In some embodiments, the LLM / VLM / MMLM, etc., disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify inappropriate or undesirable inputs (e.g., prompts) and / or outputs of the model. In doing so, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific undesirable inputs using LLM / VLM / MMLM, etc., and / or to prevent the output or presentation (e.g., displays, audio outputs, etc.) of information generated using LLM / VLM / MMLM, etc. In some embodiments, one or more additional models or layers thereof can be implemented to identify problems with the model's inputs and / or outputs. For example, these "safeguard" models can be trained to identify inputs and / or outputs that are "safe," normal, or desired for a particular application / implementation, and / or that are "unsafe" or undesirable. Therefore, the LLM / VLM / MMLM etc disclosed herein are unlikely to output language / text / audio / video / design data / USD data that may be offensive, vulgar, inappropriate, insecure, extraterritorial and / or undesirable for a particular application / implementation.

[0100] In some embodiments, LLM / VLM, etc., can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is ideally unsuitable, the model may have instructions to access one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. As another example, where at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve one or more problems, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) any number of iterations and with any number of plugins and / or APIs until a response to the input prompt that addresses each query / question / request / process / operation, etc., can be generated. Thus, one or more models may not only rely on their own training knowledge on one or more large datasets, but also on the expertise or optimization properties of one or more external resources (such as APIs, plugins, etc.).

[0101] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM, etc.), multiple instances of the same language model, and / or multiple prompts provided to the same language model or instances of the same language model can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate portions of the query. In at least one embodiment, the same input query and prompts (e.g., sets of constraints, conditions, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one prompt to constrain, guide, or otherwise influence the style, content, or characters of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be queried to provide output corresponding to different roles, perspectives, features, or different knowledge bases, as defined by the supplied prompts.

[0102] In any of these embodiments, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instance proxies of at least one language model, and / or two or more prompts provided to at least one language model can be further processed (e.g., aggregated, compared, or filtered) or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to other language models for further processing and / or validation. In one or more embodiments, language models can be queried to generate or otherwise obtain output about input source material, wherein the output is associated with the input source material. This association may include, for example, generating captions or text portions embedded (e.g., as metadata) along with the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence of a target word in an object in a text portion or image, wherein the text or image is annotated to record this presence (or absence). Alternatively, for example, but not limited to, the determination of the language model can be used to determine whether the source material should be included in the refined dataset.

[0103] Figure 8A This is a block diagram of an example generative language model system 800 applicable to implementing at least some embodiments of this disclosure. Figure 8A In the example illustrated, the generative language model system 800 includes a retrieval enhancement generation (RAG) component 892, an input processor 805, a tokenizer 810, an embedding component 820, a plug-in / API 895, and a generative language model (LM) 830 (which may include LLM, VLM, multimodal LM, etc.).

[0104] At a higher level, the input processor 805 may receive input 801, depending on the architecture of the generative LM 830 (e.g., LLM / VLM / MMLM, etc.), which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (such as OpenUSD), etc.). In some embodiments, input 801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 801 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., in tabular format, JSON, or XML). In some implementations where the generative LM 830 is capable of handling multimodal input, input 801 may combine text (or may omit text) with image data, audio data, video data, design data, USD data, and / or other types of input data (such as, but not limited to, the input data described herein). Taking raw input text as an example, input processor 805 can prepare the raw input text in various ways. For instance, input processor 805 can perform various types of text filtering to remove noise from relevant text content (e.g., special characters, punctuation, HTML tags, stop words, parts of one or more images, parts of audio, etc.). In an example involving stop words (common words that tend to carry little semantic meaning), input processor 805 can remove stop words to reduce noise and focus the generative LM 830 on more meaningful content. Input processor 805 can apply text normalization (e.g., by converting all characters to lowercase, removing accent marks, and / or handling special cases such as abbreviations or acronyms) to ensure consistency. These are just a few examples, and other types of input processing can be applied.

[0105] In some embodiments, RAG component 892 (which may include one or more RAG models and / or may be performed using the generative LM 830 itself) can be used to retrieve additional information for the portion to be used as input 801 or a prompt. The RAG can be used to enhance input to LLM / VLM / MMLM, etc., with external knowledge to make it more relevant to the answer to a specific question or query or request, such as when specific knowledge is required. RAG component 892 may obtain this additional information (e.g., grounding information, such as base text / images / videos / audio / USD / CAD, etc.) from one or more external sources, and then feed it along with the prompt to LLM / VLM / MMLM, etc., to improve the accuracy of the model's response or output.

[0106] For example, in some embodiments, in addition to the data retrieved using RAG component 892, input 801 may be generated using queries or inputs to the model (e.g., questions, requests, etc.). In some embodiments, input processor 805 may analyze input 801 and communicate with RAG component 892 (or in various embodiments, RAG component 892 may be part of input processor 805) to identify relevant text and / or other data to provide to generative LM 830 as additional context or information sources, typically identifying responses, answers, or outputs 890 from the additional context or information sources. For example, when the input indicates that the user is interested in the expected tire pressure for a particular brand and model of vehicle, RAG component 892 may, for example, perform a vector search in the embedding space using the RAG model to retrieve tire pressure information or corresponding text from a digital (embedded) version of the owner's manual for that particular brand and model of vehicle. Similarly, when a user revisits the chatbot related to a specific product system or service, the RAG component 892 can retrieve previously stored conversation history or at least its summary, and include the previous conversation history along with the current inquiry / request as part of the input 801 to the generative LM 830.

[0107] RAG component 892 can use various RAG techniques. For example, when a document is indexed, chunked, and applied to an embedding model to generate embeddings corresponding to the chunks, naive RAG can be used. User queries can also be applied to the embedding model of RAG component 892 and / or other embedding models, and the embeddings of blocks can be compared together with the embeddings of queries to identify the embeddings most similar to / most relevant to the query. These embeddings can be fed to the generative LM 830 to generate output.

[0108] In some embodiments, more advanced RAG techniques may be used. For example, the blocks may undergo a pre-retrieval process (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Additionally, a post-retrieval process (e.g., reordering, hint compression, etc.) may be performed on the output of the embedding model before the final embedding is generated and before it is used for comparison with the input query.

[0109] As another example, modular RAG techniques can be used, such as those similar to naive and / or advanced RAGs, and also include features such as hybrid search, recursive retrieval and query engines, StepBack pathways, subqueries and hypothetical document embeddings.

[0110] As another example, a graph RAG can use knowledge graphs as a source of contextual or factual information. A graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / VLM / MMLM, etc. Unlike (or other than) providing models with data blocks extracted from large documents (which can lead to a lack of context, truthfulness, linguistic accuracy, etc.), a graph RAG can also provide structured entity information to LLM / VLM / MMLM, etc., by combining structured entity text descriptions with their many attributes and relationships, allowing the model to make deeper insights. When implementing a graph RAG, the systems and methods described herein use graphs as content storage, extract relevant document blocks, and query LLM / VLM / MMLM, etc., using them to answer. In such embodiments, the knowledge graph can contain relevant textual content and metadata about the knowledge graph and can be integrated with a vector database. In some embodiments, a graph RAG can use graphs as subject matter experts, where descriptions of concepts and entities relevant to queries / hints can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, which can be executed, and LLM / VLM / MMLM, etc., can summarize the results. In this example, the graph can store relevant real-world information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tool). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from multiple avenues.

[0111] In any embodiment, RAG component 892 may implement plugins, APIs, user interfaces, and / or other functions for performing RAG. For example, LLM / VLM / MMLM, etc., can use graph RAG plugins to run queries against knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries against vector databases. For example, the graph database can interact with the plugin's REST interface, thereby decoupling the graph database from the vector database and / or the embedded model.

[0112] The tokenizer 810 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the specific implementation, these tags can represent individual words, subwords, characters, audio / video / image portions, etc. Word-based tokenization divides the text into individual words and treats each word as a separate tag. Subword tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 830 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 830 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the current task, and / or the characteristics of the training dataset. Therefore, the tokenizer 810 can convert (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0113] Embedding component 820 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 820 can use pre-trained word vectors (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, term frequency-inverse document frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or other methods.

[0114] In some implementations where input 801 contains image data / audio data, the input processor 801 may resize the image data to a standard size compatible with the corresponding input channel format and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, while the embedding component 820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 801 contains audio data, the input processor 801 may resample the audio file to a consistent sampling rate for uniform processing, while the embedding component 820 may use any known technique to extract and encode audio features—such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 801 contains video data, the input processor 801 may extract frames or resize extracted frames, while the embedding component 820 may extract features such as optical flow embedding or video embedding, and / or may encode temporal information or frame sequences. In some implementations where input 801 contains multimodal data, the embedded component 820 may use techniques such as early fusion (cascaded), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, audio, design, etc.).

[0115] Other components of the generative LM 830 and / or generative LLM system 800 can use different types of neural network architectures depending on the specific implementation. For example, transformer-based architectures (such as those used in models like GPT) can be implemented and can include self-attention mechanisms that measure the importance of different words or tokens in the input sequence and / or feedforward networks that process the output of the self-attention layers, apply nonlinear transformations to the input representation, and extract higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of architectural adversarial networks (e.g., generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation and architecture, the embedded component 820 can apply the encoded representation of input 801 to the generative LM 830, and the generative LM 830 can process the encoded representation of input 801 to generate output 890, which may include response text and / or other types of data.

[0116] As described herein, in some embodiments, the generative LM 830 may be configured to access or use (or be able to access or use) plugins / APIs 895 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 830 is not ideally suited, the model may have instructions for accessing one or more plugins / APIs 895 (e.g., third-party plugins) (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 892) to help process the current input. In such an example, if at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 895 to the plugin / API 895, which can process the information and return an answer to the generative LM 830, which can then use the response to generate output 890. This process can be repeated, for example, recursively iterated an arbitrary number of times using any number of plugins / APIs 895, until an output 890 can be generated that solves each query / question / request / procedure / action / etc. from input 801. Therefore, the model can rely not only on its own knowledge derived from training on a large dataset and / or data retrieved using the RAG component 892, but also on the expertise or optimized features of one or more external resources (such as plugins / APIs 895).

[0117] Figure 8B This is a block diagram of an example implementation of a generative LM 830 including a transducer encoder-decoder. For example, suppose input text such as "Who discovered gravity" is tokenized (e.g., via...). Figure 8A The tokenizer 810 in the middle is a tokenizer for tokens such as words, and each token is encoded (e.g., by...). Figure 8A The embedding component 820 in the input sequence is the corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Therefore, the (e.g., generated) embeddings can be applied to one or more encoders 835 of the generative LM 830.

[0118] In one example implementation, encoder 835 forms an encoder stack, where each encoder contains a self-attention layer and a feedforward network. In the example transformer architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a series of vectors, passing each vector through the self-attention layer, then the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for the token pair is computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying it by the corresponding value vector, and summing the weighted value vectors. Encoders can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector that encodes the input. Attention projection layer 840 can transform the context vector into attention vectors (keys and values) for decoder 845.

[0119] In the example implementation, decoder 845 constitutes a decoder stack, where each decoder contains a self-attention layer, an encoder-decoder self-attention layer (which uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence), and a feedforward network. Similar to encoder 835, in the example transformer architecture, each token (e.g., a word) flows through a separate path in decoder 845. During the first pass, decoder 845, classifier 850, and generation mechanism 855 can generate a first token, and generation mechanism 855 can apply the generated token as input during the second pass. This process can be repeated cyclically, continuously generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes, applying the token embedding and positional encoding of the composite sequence as input to decoder 845, generating one token at a time (called autoregression) until a symbol or token representing the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before softmax computation. In one example implementation, the encoder-decoder attention layer operates similarly to (e.g., multi-head) self-attention in encoder 835, except that it creates its query from the layer below it and obtains keys and values ​​(e.g., matrices) from the output of encoder 835.

[0120] Therefore, decoder 845 can output some decoded (e.g., vector) representation of the input applied during a specific iteration. Classifier 850 can include a multi-class classifier comprising one or more neural network layers (which project the decoded (e.g., vector) representations onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary)) and a softmax computation that converts unnormalized log probabilities into probabilities. Thus, generation mechanism 855 can select or sample words or tokens based on their corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append them to the output of the previous round, thereby sequentially generating each word or token. Generation mechanism 855 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token indicating the end of the response is selected or sampled, at which point generation mechanism 855 can output the generated response.

[0121] Figure 8C This is a block diagram of an example implementation of a generative LM 830 converter architecture that includes a decoder-only converter. For example, Figure 8C The operation mode of the decoder 860 in the middle and Figure 8B The decoder in the 845 is similar, except... Figure 8C Each decoder 860 in the implementation omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, the decoders 860 can form a decoder stack, where each decoder contains a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, symbols or tags representing the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to the decoder 860. Figure 8B Similar to decoder 845, each tag (e.g., a word) can flow through a separate path in decoder 860, and decoder 860, classifier 865, and generation mechanism 870 can use autoregression to sequentially generate tags one at a time until a symbol or tag indicating the end of the response is predicted. The operation of classifier 865 and generation mechanism 870 can be similar to... Figure 8B The classifier 850 and generation mechanism 855 are similar, wherein the generation mechanism 870 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous round, thereby sequentially generating each label until a symbol or label indicating the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0122] Figure 9 This is a block diagram of a computing system 900 having two processing devices coupled to each other and multiple networks according to at least one embodiment. The computing system 900 is designed with multiple integrated circuits (referred to as processing devices), each including a CPU and two GPUs, forming a powerful and flexible architecture. These processing devices are interconnected via NVLink (or other high-speed interconnects) to enable high-speed communication between the processing devices, and are also connected via network interface cards (NICs) or data processing units (DPUs) to ensure efficient data transfer across the computing system 900. The processing devices coupled via NVLink enable seamless data exchange and parallel processing, thereby enhancing overall computing performance. Additionally, these processing devices are connected to multiple networks via one or more network interface cards (NICs) or DPUs, enabling the system to handle complex multi-network tasks with high bandwidth and low latency. This configuration makes the computing system 900 ideal for demanding applications requiring significant processing power, such as artificial intelligence (AI), machine learning (ML), and data-intensive computing, while ensuring robust connectivity and scalability across various networking environments. The integrated circuits of the computing system 900 may include one or more CPUs and one or more GPUs. Figure 9 The diagram illustrates an example architecture for a multi-GPU architecture.

[0123] like Figure 9 As illustrated, the computing system 900 includes a processing device 902 with a multi-GPU architecture. Specifically, the processing device 902 includes a CPU 906, a GPU 908, and a GPU 910. The CPU 906 can be coupled to the GPU 908 via a chip-to-chip (D2D) or chip-to-chip (C2C) interconnect 912 (such as a Ground Reference Signaling Interconnect (GRS interconnect)). The CPU 906 can be coupled to the GPU 910 via a D2D or C2C interconnect 914. The CPU 906 can also be coupled to the GPU 908 and GPU 910 via a PCIe interconnect. The CPU 906 can be coupled to one or more network interface cards (NICs) or data processing units (DPUs), which in turn are coupled to one or more networks. For example, as... Figure 9 As illustrated, CPU 906 is coupled to a first NIC / DPU 926, which in turn is coupled to network 930. CPU 906 is also coupled to a second NIC / DPU 928, which is also coupled to network 930. NIC / DPU 926 and NIC / DPU 928 can be coupled to network 930 via Ethernet (ETH) or InfiniBand (IB) connections.

[0124] The computing system 900 also includes a processing device 904 with a multi-GPU architecture. Specifically, the processing device 904 includes a CPU 916, a GPU 918, and a GPU 920. The CPU 916 can be coupled to the GPU 918 via a D2D or C2C interconnect 922. The CPU 916 can be coupled to the GPU 920 via a D2D or C2C interconnect 924. The CPU 916 can also be coupled to the GPU 918 and GPU 920 via a PCIe interconnect. The CPU 916 can be coupled to one or more NICs or DPUs, which in turn are coupled to one or more networks. For example, as... Figure 9 As illustrated, CPU 916 is coupled to a first NIC / DPU 932, which in turn is coupled to network 936. CPU 916 is also coupled to a second NIC / DPU 934, which is also coupled to network 936. NIC / DPU 932 and NIC / DPU 934 can be coupled to network 936 via Ethernet (ETH) or InfiniBand (IB) connections.

[0125] In at least one embodiment, processing device 902 and processing device 904 can communicate with each other via NIC / DPU 938 (such as via PCIe interconnect). Processing device 902 and processing device 904 can also communicate with each other via high-bandwidth communication interconnect 940 (such as NVLink interconnect or other high-speed interconnect).

[0126] In at least one embodiment, the computing system 900 is used for high-speed network communication and includes a processing unit (e.g., CPU 906, GPU 908, GPU 910, CPU 916, GPU 918, GPU 920, NIC / DPU 926, NIC / DPU 928, NIC / DPU 932, NIC / DPU 934, or NIC / DPU 938) and a network interface coupled to the processing unit. The processing unit and network interface can be used to employ multilingual speech-to-text models and language indicators (such as by executing...). Figure 4 The pipeline for multilingual automatic speech recognition is implemented by operating and training various machine learning models (e.g., STT model, LM model, etc.).

[0127] Figure 10 This is a block diagram of a computing system 1000 having a CPU 1002 and a GPU 1004 on a single integrated circuit, according to at least one embodiment. The computing system 1000 can be a highly integrated design in which the CPU 1002 and GPU 1004 are connected on a single integrated circuit, thereby enabling fast, low-latency communication between the two processing units using NVLink C2C (chip-to-chip) interconnect 1006. This tight integration allows for efficient data transfer and parallel processing between the CPU 1002 and GPU 1004, thereby optimizing the execution of complex computational tasks. The GPU elements within the computing system 1000 can be interconnected using an NVLink network, allowing for scalability of up to 256 GPU elements, creating a powerful, unified, and ideal processing environment for large-scale AI, ML, and high-performance computing applications. The NVLink network can be a GPU configuration of a high-bandwidth communication interconnect 1010. Additionally, the computing system 1000 can be designed to connect to a high-speed I / O interface via a PCIe interconnect 1008, thereby ensuring rapid data transfer with external devices, further enhancing the system's ability to handle data-intensive tasks, and providing robust connectivity to peripheral components. It should be noted that since the CPU 1002 and GPU 1004 reside on the same integrated circuit, the C2C interconnect 1006 can be considered a D2D interconnect. The integrated circuit may include CPU memory (also referred to as main memory) and GPU memory, which the CPU 1002 and GPU 1004 can access separately via the high-speed interconnect. The computing system 1000 can combine the performance of the GPU 1004 with the versatility of the CPU 1002. The CPU 1002 can be connected to a high-bandwidth and memory-coherent C2C interconnect 1006 within a single integrated circuit. The computing system 1000 can support a link-switching system.

[0128] In at least one embodiment, the computing system 1000 is used for high-speed network communication and includes a processing unit (e.g., CPU 1002, GPU 1004, NVLink network) and a network interface coupled to the processing unit. The processing unit and network interface can be used to utilize multilingual speech-to-text models and language indicators (such as by executing...). Figure 4 The pipeline for multilingual automatic speech recognition is implemented by operating and training various machine learning models (e.g., STT model, LM model, etc.).

[0129] Figure 11 This is a block diagram of a computing system 1100 having a Tensor Core GPU 1108 according to at least one embodiment. The computing system 1100 may be a DGX H100 system, which is a high-performance computing platform designed to meet the requirements of AL, ML, and deep learning (DL) workloads. The computing system 1100 may include multiple Tensor Core GPUs 1108 (e.g., NVIDIA H100 Tensor Core GPUs). Each of the Tensor Core GPUs 1108 may be one of the above-described... Figure 10 One of the integrated circuits described is the Tensor Core GPU 1108, which is optimized for AI / ML / DL applications, delivering superior performance for deep learning training, inference, and high-performance computing tasks. The Tensor Core GPUs 1108 within the computing system 1100 interconnect using high-speed communication interfaces such as NVLink, enabling rapid data transfer between them, crucial for handling large-scale AI models and datasets with low latency. The computing system 1100 is designed for scalability, allowing for the integration of additional GPUs as needed, making it sufficiently versatile for research, development, and deployment in data centers used for production AI workloads. Each GPU is equipped with Tensor Cores, dedicated processing units that accelerate matrix operations and are fundamental components of AI and deep learning algorithms. These Tensor Cores enable the system to perform mixed-precision computations efficiently, balancing speed and accuracy. Considering the power consumption and heat generation of multiple Tensor Core GPUs 1108, the computing system 1100 may include advanced cooling solutions and power management features to ensure safe operation while maintaining peak performance. It is supported by a comprehensive software ecosystem that includes NVIDIA's CUDA programming model, AI frameworks such as TensorFlow and PyTorch, and other HPC and AI software tools that enable developers and researchers to fully leverage the TensorCore GPU 1108 for their specific applications. The Computing System 1100 is ideally suited for large-scale AI model training, real-time inference, scientific simulations, data analysis, and other compute-intensive tasks requiring significant parallel processing power.

[0130] The Tensor Core GPU 1108 can be coupled to multiple CPUs, such as CPU 1102 and CPU 1104, using a switch 1106 (e.g., a CX7 HCA / NIC with a PCIe switch). The Tensor Core GPU 1108 can be coupled to each other via a switch 1110 (e.g., an NVSwitch). Switches 1106 and 1110 can be coupled to a high-speed transceiver module 1112. The high-speed transceiver module 1112 can be an eight-channel small form factor pluggable (OSFP) module. OSFP modules are high-speed transceiver modules designed for fast data communication (specifically, in environments requiring high bandwidth, such as data centers and high-performance computing systems). These modules support extremely high data rates, typically up to 400 Gbps per module, with future capabilities expanding to 800 Gbps or higher. OSFP modules interface with the system via a PCIe interface, enabling fast and efficient data transfer between the integrated CPU-GPU components and external networks or other connected systems. Their hot-swappable nature allows for easy insertion or removal without shutting down the system, providing flexibility and maintainability crucial in critical uptime environments. Additionally, OSFP modules are designed for high density, maximizing the number of high-speed connections within limited spaces, such as dense server racks. By complying with the latest networking standards, OSFP modules ensure that the Computing System 1100 can meet ever-increasing data demands and can be upgraded to support future advancements in network speeds, thus contributing to improved overall system performance and scalability.

[0131] In at least one embodiment, the computing system 1100 can be viewed as a data network configuration with full-bandwidth in-server NVLink. In this example, all eight tensor core GPUs 1108 can simultaneously saturate eighteen NVLinks within the server leading to other GPUs. Bandwidth is limited by the over-subscription of multiple other GPUs. In another embodiment, the data network configuration can be half-bandwidth in-server NVLink. In this example, all eight tensor core GPUs 1108 can half-subscribe to eighteen NVLinks leading to GPUs in other servers. Four tensor core GPUs 1108 can saturate eighteen NVLinks leading to GPUs in other servers. This is equivalent to full bandwidth on AllReduce with Scalable Hierarchical Aggregation and Reduction Protocol (SHARP). The reduction in all-to-all (All2All) bandwidth is a trade-off between server complexity and cost. In at least one embodiment, all eight tensor core GPUs 1108 can independently transfer data in a multi-track InfiniBand / Ethernet configuration using the Remote Direct Memory Access (RDMA) protocol via their own dedicated switches (e.g., 400Gb / s HCA / NIC). In this example, for non-NVLink network devices, the total aggregated full-duplex bandwidth is 800 GBp.

[0132] In at least one embodiment, the computing system 1100 is used for high-speed network communication and includes a processing unit (e.g., CPU 1102, switch 1106, Tensor Core GPU 1108, switch 1110, high-speed transceiver module 1112) and a network interface coupled to the processing unit. The processing unit and network interface can be used to employ multilingual speech-to-text models and language indicators (such as by executing...). Figure 4 The pipeline for multilingual automatic speech recognition is implemented by operating and training various machine learning models (e.g., STT model, LM model, etc.).

[0133] Other variations are also within the spirit of this disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments are shown in the accompanying drawings and have been described in detail above. However, it should be understood that this disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined by the appended claims.

[0134] In the context of describing the disclosed embodiments (especially in the context of the following claims), unless otherwise indicated herein or obviously contradicted by the context, the use of the terms "a," "an," and "the," and similar designations, should be interpreted as encompassing both the singular and plural, and not as definitions of the terms. Unless otherwise indicated, the terms "comprising," "having," "including," and "containing" should be interpreted as open-ended terms (meaning "including but not limited to"). When unmodified and referring to a physical connection, "connection" should be interpreted as partially or completely contained in, attached to, or combined with, even with intervening elements. Unless otherwise indicated herein, statements of value ranges herein are intended only as a way of referring to each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually stated herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms "set" (e.g., "item set") or "subset" should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise specified or contradicted by the context, the term "subset" of the corresponding set does not necessarily refer to an appropriate subset of the corresponding set, but rather the subset and the corresponding set can be equal.

[0135] Unless explicitly stated otherwise or otherwise clearly contradicted by the context, connectives (such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C") are otherwise understood, together with the context, to generally refer to items, terms, etc., and can be any non-empty subset of the set A or B or C, or A and B and C. For example, in an illustrative example of a set with three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connectives are generally not intended to imply that certain embodiments require the separate presence of at least one of A, at least one of B, and at least one of C. Additionally, unless explicitly stated or contradicted by the context, the term "multiple" indicates a plural state (e.g., "multiple items" indicates multiple items). In at least one embodiment, the number of items in the multiple is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase "based on" means "at least partially based on" rather than "based on only".

[0136] Unless otherwise indicated herein or otherwise clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include transient signals (e.g., propagation of transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver that includes transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored in a set of one or more non-transitory computer-readable storage media having executable instructions (or other memory for storing executable instructions) stored thereon, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more individual non-transitory storage media lack the complete code, instead the multiple non-transitory computer-readable storage media collectively store the complete code. In at least one embodiment, the executable instructions are configured such that different instructions are executed by different processors.

[0137] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the performance of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, while in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0138] Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of this disclosure and is not intended to limit the scope of this disclosure. The language in the specification should not be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0139] All references cited in this document (including publications, patent applications and patents) are incorporated herein by reference to the same extent that each reference is individually and specifically indicated to be incorporated herein by reference and fully elaborated herein.

[0140] In the specification and claims, the terms "coupled" and "connected" along with their derivatives may be used. It should be understood that these terms are not intended to be synonyms with each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0141] Unless otherwise specifically stated, it should be understood that throughout this specification, terms such as “processing,” “calculating,” “operating,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data representing physical (such as electronic) quantities within the registers and / or memory of the computing system into other data representing physical quantities within the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0142] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and transforms that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a network device. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that the system can embody one or more methods, and the method can be considered a system.

[0143] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer implementation machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter of a function call, an application programming interface, or an inter-process communication mechanism.

[0144] While the discussion herein illustrates exemplary embodiments of the described technologies, other architectures may also be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for discussion purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0145] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms for implementing the claims. < / eos> < / sos> < / eos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A method, the method comprising: Using a multilingual speech-to-text (STT) model of a multilingual automatic speech recognition (ASR) system and using audio samples as input to the multilingual STT model, determine the text transcription associated with the audio samples and one or more language indicators associated with corresponding grammatical units in one or more grammatical units of the text transcription; Using the language indicators from the one or more language indicators to identify the monolingual language model LM among the multiple monolingual language models LM of the automatic speech recognition (ASR) system; and This enables the use of the identified language model LM and a subset of the text transcription as input to the identified language model LM to optimize the text transcription associated with the audio sample.

2. The method of claim 1, further comprising: Remove one or more language indicators from the text transcription.

3. The method of claim 1, further comprising: The multilingual speech-to-text (STT) model is trained using training data, which includes one or more second audio samples, one or more second text transcriptions each associated with a corresponding audio sample in the one or more second audio samples, and one or more second language indicators each associated with a corresponding text transcription in the one or more second text transcriptions.

4. The method of claim 1, wherein, The one or more grammatical units of the text transcription are sentences, and wherein the one or more language indicators are each located after the punctuation marks of the respective sentences.

5. The method of claim 1, wherein, The multilingual vocabulary of the multilingual speech-to-text (STT) model includes one or more language indicators.

6. The method of claim 1, wherein, The model architecture of the multilingual speech-to-text (STT) model corresponds to that of the monolingual STT model.

7. The method of claim 1, wherein, Two of the one or more language indicators are each associated with a code-switching grammar unit of the one or more grammar units of the text transcription.

8. The method of claim 7, wherein, The code-switching syntax unit is a sentence, and the two language indicators are located after the punctuation marks of the sentence.

9. A system comprising: One or more processors are configured to perform operations, said operations including: At least based on a multilingual speech-to-text (STT) model, audio data corresponding to an audio sample is processed to determine the text transcription associated with the audio sample and one or more language indicators associated with a corresponding grammatical unit in one or more grammatical units of the text transcription. At least a portion of the transcription and associated language indicators from the one or more language indicators are processed based on a monolingual language model (LM) to determine an updated text transcription; and One or more operations are performed using the updated text transcription.

10. The system of claim 9, wherein the operation further comprises: Before performing the one or more operations, remove the one or more language indicators from the text transcription.

11. The system of claim 9, wherein the operation further comprises: The multilingual speech-to-text (STT) model is trained using training data, which includes one or more second audio samples, one or more second text transcriptions each associated with a corresponding audio sample in the one or more second audio samples, and one or more second language indicators each associated with a corresponding text transcription in the one or more second text transcriptions.

12. The system of claim 9, wherein, The one or more operations include at least one of the following: This enables the use of one or more display devices of the system to present at least a portion of the updated text transcription; Translate at least a portion of the updated text transcription into other languages, and display it using one or more display devices; or At least a portion of the updated text transcription is translated into another language, and audio output is performed using synthesized sounds corresponding to said portion of the updated text transcription.

13. The system of claim 9, wherein, The multilingual vocabulary of the multilingual speech-to-text (STT) model includes one or more language indicators.

14. The system of claim 9, wherein, The model architecture of the multilingual speech-to-text (STT) model corresponds to that of the monolingual STT model.

15. One or more processors, said one or more processors including processing circuitry for causing to perform an operation, said operation including: The method involves processing an audio segment using at least a multilingual language model to generate an initial text transcription including one or more language indicators, and processing the initial text transcription and the one or more language indicators using one or more monolingual language models to generate a final text transcription, thereby generating the final text transcription corresponding to the audio segment.

16. The one or more processors as claimed in claim 15, wherein the operation further comprises: During the generation of the final text transcription, one or more language indicators are removed from the initial text transcription.

17. The processor of claim 15 or more, wherein the operation further comprises: The multilingual language model is trained using training data, which includes one or more second audio segments, one or more second text transcriptions each associated with a corresponding audio segment in the one or more second audio segments, and one or more second language indicators each associated with a corresponding text transcription in the one or more second text transcriptions.

18. One or more processors as claimed in claim 15, wherein, The final text transcription is one of the following: stored on the device, visually presented on the device, or used to generate synthetic audio output using the device.

19. One or more processors as claimed in claim 15, wherein, The initial text transcription includes two different language indicators corresponding to code switching within the audio segment, and the final text transcription is generated using a first monolingual language model corresponding to the first language in the initial text transcription and a second monolingual language model corresponding to the second language in the initial text transcription.

20. One or more processors as claimed in claim 15, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; A system that includes one or more machine learning models deployed in services or microservices and OS-level virtualization packages (e.g., containers); A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images