PICTURE TABLES FOR APPLICATION PROGRAMMING INTERFACES CALLED BY NEURAL NETWORK AGENTS

Abstract

Devices, systems, and techniques for mapping tokens generated by neural networks to application programming interfaces (APIs). In at least one embodiment, a processor includes one or more circuits that cause one or more tables to map one or more tokens generated by one or more neural networks to one or more tokens that are to be used by one or more APIs.

Description

TECHNICAL AREA

[0001] At least one embodiment involves processing resources used to run neural networks whose output can invoke application programming interfaces (APIs). For example, at least one embodiment involves processors or computing systems that cause tables to map tokens generated by neural networks to tokens used as API inputs. BACKGROUND

[0002] Machine learning models based on neural networks, such as Large Language Models (LLMs), are being used for increasingly important tasks. In some cases, LLM agents use external tools by calling APIs. Techniques to ensure that LLM agents call APIs correctly can be improved. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a logical block diagram of an exemplary system architecture in which mapping tables can be used to guide the generation of output for calling APIs by neural networks, according to at least one embodiment; Fig. 2 illustrates exemplary data structures that can be used to generate output for calling APIs by neural networks, according to at least one embodiment; Fig. Figure 3 illustrates an example of converting a regular expression indicating an acceptable or valid API input into a representation of a finite state machine (FSM) according to at least one embodiment; Fig. Figure 4 illustrates an example of a workflow for mapping tokens of a neural network vocabulary to transition identifiers of an FSM representation of an acceptable or valid API input according to at least one embodiment; Fig. Figure 5 illustrates an exemplary creation of a table that maps current API states to next API states based on bytes of API input, according to at least one embodiment; Fig. Figure 6 illustrates an exemplary creation of a table that maps current API states to next API states based on tokens generated by a neural network, according to at least one embodiment; Fig. Figure 7 illustrates an exemplary inference time modification, using one or more mapping tables, of output probabilities assigned to tokens on a neural network, according to at least one embodiment; Fig. Figure 8 illustrates a method for generating and using mapping tables for calling APIs of LLM agents according to at least one embodiment; Fig. Figure 9 illustrates an exemplary system in which mapping tables for APIs can be dynamically generated and cached for reuse with resources of an inference service, according to at least one embodiment; Fig. Figure 10 illustrates an exemplary data center system according to at least one embodiment. Fig. Figure 11 illustrates a system-on-a-chip (SOC) according to at least one embodiment; Fig. Figure 12A illustrates a parallel processor according to at least one embodiment; Fig. Figure 12B illustrates a processing cluster according to at least one embodiment; Fig. Figure 12C illustrates a graphics multiprocessor according to at least one embodiment; Fig. Figure 13 illustrates an accelerator processor according to at least one embodiment; Fig. Figure 14A illustrates a central processing unit and a core of the central processing unit according to at least one embodiment; Fig. Figure 14B illustrates a core of the central processing unit in Fig. 14A according to at least one embodiment; Fig. Figure 15 illustrates another accelerator processor according to at least one embodiment; Fig. Figure 16 illustrates a neuromorphic processor according to at least one embodiment; Fig. 17 illustrates a supercomputer according to at least one embodiment; Fig. Figure 18 illustrates another accelerator processor according to at least one embodiment; Fig. 19 illustrates another processor according to at least one embodiment; Fig. Figure 20 illustrates another accelerator processor according to at least one embodiment; Fig. 21 illustrates a tensor processing unit according to at least one embodiment; Fig. Figure 22 illustrates a RISC-V compatible processor according to at least one embodiment; Fig. 23A and Fig. Figure 23B illustrates a speech processing unit according to at least one embodiment; Fig. Figure 24 illustrates a software stack of a programming platform according to at least one embodiment. Fig. Figure 25 illustrates software supported by a programming platform, according to at least one embodiment. Fig. 26 illustrates compiling code for execution on programming platforms of Fig. 18 according to at least one embodiment; Fig. 27 illustrates an example of an autonomous vehicle and its system architecture according to at least one embodiment; Fig. 28A illustrates inference and / or training logic according to at least one embodiment. Fig. 28B illustrates inference and / or training logic according to at least one embodiment. Fig. 28C illustrates the training and deployment of a neural network according to at least one embodiment. DETAILED DESCRIPTION

[0003] Fig. Figure 1 illustrates a logical block diagram of an exemplary system architecture in which mapping tables can be used to guide the generation of output for calling APIs by neural networks, according to at least one embodiment. In at least one embodiment, language model 150, which includes one or more neural networks, such as neural network 172, can be executed in a language model agent execution environment 152, which includes one or more processors. In at least one embodiment, language model 150 can be executed to prepare responses, such as response 162, to input prompts, such as input prompt 161. In at least one embodiment, output generated by language model 150 to prepare response 162 can include one or more language model vocabulary tokens 133, to which respective token identifiers can be assigned by tokenizer 135.

[0004] In at least one embodiment, to prepare response 162, one or more external tools, such as external tool 144A and / or external tool 144B, can be called by language model 150 or by an associated Large Language Model (LLM) agent that has language model 150. In at least one embodiment, an external tool such as external tool 144A or external tool 144B can implement or expose one or more application programming interfaces (APIs) that can be called based at least partially on output generated by one or more neural networks such as neural network 172.In at least one embodiment, individual external tools whose APIs can be called, based at least partially on neural network output, can comprise respective sets of valid or acceptable API syntax or signature tokens that represent constraints on input to be obtained by the external tools to ensure correct functionality of the external tools. For example, in at least one embodiment, external tool 144A can comprise associated acceptable API syntax / signature tokens 156A, and external tool 144B can comprise associated acceptable API syntax / signature tokens 156B. In at least one embodiment, for example, external tool 144A can include a flight ticket reservation system that can be used to respond to input prompts 161 relating to travel reservations.In at least one embodiment, for example, external tool 144B can include a restaurant application that can be used to respond to input prompts 161 relating to restaurant reservations or menu-related queries. In at least one embodiment, for example, an input prompt 161 can result in calls to multiple APIs of one or more external tools.

[0005] In at least one embodiment, if output generated by one or more neural networks 172 for API calls made in response to input prompt 161 does not comply with input restrictions corresponding to acceptable API syntax / signature tokens, this can lead to errors in preparing response 162. In at least one embodiment, language model 150 and / or an LLM agent that uses or has language model 150 can include an associated API Restriction Compliance Manager 164, which is responsible for directing the generation of output by neural network 172 to comply with restrictions on acceptable API syntax or signatures.In at least one embodiment, API Restriction Compliance Manager 164 can generate one or more mapping tables 174, which are used to ensure that output generated by one or more neural networks 172 for API calls, such as API calls 155A or API calls 155B, satisfies input restrictions that comply with acceptable API syntax / signature tokens 156A or acceptable API syntax / signature tokens 156B. In at least one embodiment, one or more mapping tables 174 can be generated by API Restriction Compliance Manager 164 to implement, at least partially, a technique referred to as guided decoding or constrained decoding for neural networks.In at least one embodiment, a workflow for generating a mapping table 174 can comprise several steps, at least some of which can be performed in parallel using multiple threads. In at least one embodiment, a workflow for generating a mapping table 174 can comprise several steps, at least some of which can be performed using multiple threads of a graphics processing unit (GPU) or a machine learning accelerator. In at least one embodiment, the language model agent execution environment 152 can include a processor that has one or more circuits to cause one or more tables to map one or more tokens, generated by one or more neural networks 172, into one or more tokens that can be used by one or more APIs of tools, such as external tool 144A or external tool 144B.

[0006] In at least one embodiment, one or more mapping tables used to generate tokens for APIs can be created prior to at least some inference operations that require calling APIs. For example, in at least one embodiment, one or more mapping tables used to generate tokens for a single API of a single external tool can be created at a time when a request to call the single external tool is determined in language model execution environment 152. For example, in at least one embodiment, one or more mapping tables used to generate tokens for a single API of a single external tool can be created in response to a determination that the single API should be added to a set of APIs called by neural networks 172.In at least one embodiment, one or more mapping tables used to generate tokens for a single API of a single external tool can be stored in a cache in language model execution environment 152 for potential reuse during inference operations.

[0007] In at least one embodiment, an entry within a single table created by API Restriction Compliance Manager 164 can indicate, given a single state of a state representation (such as a finite state machine or FSM) of accepted or valid input tokens generated by one or more neural networks for a single API, whether the generation of a single token by the one or more neural networks results in input that is not acceptable for the single API.In at least one embodiment, an entry within a single table created by API Restriction Compliance Manager 164 can indicate, given a single state of a state representation of accepted or valid input tokens generated by one or more neural networks for a single API, a next state of the state representation that will be reached when generation of a single token by the one or more neural networks results in input that is acceptable for the single API.In at least one embodiment, respective portions of a single table mapping neural network vocabulary tokens to API FSM states can be created at least partially in parallel by API Constraint Compliance Manager 164 by assigning respective threads to a plurality of threads to generate entries of the respective portions for respective subsets of a vocabulary of tokens of the one or more neural networks. In at least one embodiment, respective portions of a single table mapping neural network vocabulary tokens to API FSM states can be created at least partially in parallel by API Constraint Compliance Manager 164 by assigning respective threads to a plurality of threads of a GPU or machine learning accelerator to generate entries of the respective portions for respective subsets of a vocabulary of tokens of the one or more neural networks.

[0008] In at least one embodiment, during inference operations performed by one or more neural networks 172, an output probability assigned to a single token can be set to zero within a neural network, based at least partially on examining (a) a single mapping table generated by API constraint compliance managers 164, and (b) a current state of a state representation of accepted input tokens of a single API. In at least one embodiment, during inference operations performed by one or more neural networks 172, an output probability assigned to a single token can be set to zero within a neural network by setting a corresponding logit value to negative infinity.In at least one embodiment, during inference operations performed by one or more neural networks 172, an output probability assigned to a single token can be set to a very small value close to zero within a neural network by setting a corresponding logit value to a very large negative number.

[0009] In at least one embodiment, one or more mapping tables can be generated by API constraint compliance manager 164 at least partially based on analysis of a first vector and a second vector, wherein the first vector comprises a list of bytes generated by converting tokens of a token vocabulary of a language model comprising one or more neural networks into corresponding bytes, and the second vector comprises start indices of individual tokens within the first vector. In at least one embodiment, analysis of the first vector and / or the second vector can be performed at least partially in parallel, e.g., by assigning respective threads to a plurality of threads to analyze respective subsets of the first vector and / or the second vector.In at least one embodiment, analysis of the first vector and / or the second vector can be performed at least partially in parallel by assigning respective threads to a plurality of threads of a GPU or machine learning accelerator to analyze respective subsets of the first vector and / or the second vector. In at least one embodiment, analysis of the first vector and / or the second vector can be performed at least partially in parallel using a parallel programming kernel such as a CUDA (Compute Unified Device Architecture) kernel.

[0010] In at least one embodiment, at least one table can be generated by API Restriction Compliance Manager 164 by mapping the byte representations of accepted input tokens of one or more APIs to the respective transition identifiers of a state transition representation (such as an FSM). In at least one embodiment, mapping the byte representations of accepted input tokens to the transition identifiers of a state representation of accepted input tokens can be performed at least partially in parallel, e.g., by assigning respective threads to a plurality of threads to map respective subsets of the byte representations.In at least one embodiment, mapping byte representations of accepted input tokens to transition identifiers of a state representation of accepted input tokens can be performed at least partially in parallel by assigning respective threads to a plurality of threads of a GPU or a machine learning accelerator to map respective subsets of the byte representations. In at least one embodiment, mapping byte representations of accepted input tokens to transition identifiers of a state representation of accepted input tokens can be performed at least partially in parallel using a parallel programming kernel, such as a CUDA kernel.

[0011] In at least one embodiment, a text representation of accepted input tokens of a single API, received in a language model execution environment via a programmatic interface, can be transformed into a state representation of accepted or valid input tokens of the single API, and the state representation can be used to generate at least one mapping table. In at least one embodiment, a text representation of accepted input tokens of a single API, used by the API Restriction Compliance Manager 164 to generate one or more mapping tables, can include a regular expression and / or a JSON (JavaScript Object Notation) descriptor, file, or object.In at least one embodiment, a text representation of accepted input tokens of a single API, used by the API Restriction Compliance Manager 164 to generate one or more mapping tables, can be received through a user-friendly interface that provides a set of primitives for specifying regular expressions, including, but not limited to, a regular expression parser (RegexParser), a union parser for combining RegexParsers using a logical OR operator, a sequence parser for combining RegexParsers in sequence, a string parser for creating a regular expression that matches a specified string, an exclusion string parser that matches any string except a string containing a specified forbidden string, and / or a JSON parser that creates a specific JSON schema.

[0012] Fig. Figure 2 illustrates exemplary data structures that can be used to generate output for calling APIs by neural networks, according to at least one embodiment. In at least one embodiment, one or more data structures, shown in Figure 2, can be used. Fig. 2, generated by API Restriction Compliance Manager 164 to guide or restrict neural networks to produce tokens or token sequences valid for calling APIs. In at least one embodiment, to facilitate the guidance of neural networks to produce tokens or token sequences valid for calling APIs, tokens of a neural network vocabulary can be converted into their respective byte-level representations, and acceptable API input tokens can also be converted into their respective byte-level representations, so that both neural network output and API input can be processed using a common representation.

[0013] In at least one embodiment, acceptable API input descriptor 202, which can be used in a workflow for generating valid API call tokens or token sequences by one or more neural networks, can contain text indicating valid sequences of tokens or characters that can be used to call an API. In at least one embodiment, acceptable API input descriptor 202 can contain one or more regular expressions. In at least one embodiment, acceptable API input descriptor 202 can contain one or more JSON schemas, objects, descriptors, or files. In at least one embodiment, acceptable API input descriptor 202 can display an alphabet of characters that can be used individually or in combination as valid input tokens by one or more APIs to be called using neural networks.In at least one embodiment, acceptable API input descriptor 202 can be obtained via one or more programmatic interfaces of a language model execution environment and used by the API constraint compliance manager 164 to prepare one or more tables that map tokens generated by neural networks to tokens consumed or used by APIs.

[0014] In at least one embodiment, acceptable API input descriptor 202 can be converted or translated into API input state representation 206 for use in subsequent stages of a workflow for generating valid API call output by neural networks. In at least one embodiment, API input state representation 206 can include a finite state machine (FSM). In at least one embodiment, API input state representation 206 can include a finite state machine (FSM) represented as a graph whose nodes represent states and whose edges represent state transitions resulting from receiving or processing a given next input token or alphabet character.

[0015] In at least one embodiment, API input state representation 206 can be converted into a byte-level API input state representation 208 (e.g., a byte-level FSM) by converting individual tokens or characters of API input state representation 206 into their constituent bytes and labels of state transitions corresponding to individual bytes. In at least one embodiment, byte-level API input state representation 208 can include a finite state machine (FSM) represented as a graph whose nodes represent states and whose edges represent state transitions resulting from a given next byte being received as input or being processed.

[0016] In at least one embodiment, an API input state-to-state transition matrix 210 can be generated using at least a byte-level API input state representation 208 (e.g., a byte-level FSM). In at least one embodiment, the API input state-to-state mapping table 210 can have a plurality of entries, each of which can map a current state of API input and a next API input byte to a next state of API input. In at least one embodiment, the API input state-to-state mapping table 210 can have a two-dimensional representation of the byte-level API input state representation 208.In at least one embodiment, API input state-to-state mapping table 210 can have a two-dimensional representation of byte-level API input state representation 208, which is more suitable as input for parallel processing kernels or code than a traditional representation of an FSM, which, for example, may require a left-to-right traverse through at least a portion of a graph to determine a next state.

[0017] In at least one embodiment, a language model comprising one or more neural networks can include an associated language model vocabulary 252, which has a collection of tokens. In at least one embodiment, individual tokens of a language model vocabulary can include words, parts of words, combinations of words, symbols, characters, and the like. In at least one embodiment, a language model vocabulary can have one or several hundred thousand tokens that the neural network or networks of the language model have learned to generate during pre-training, fine-tuning, and / or other training phases.

[0018] In at least one embodiment, the byte-level vocabulary representation vector set 254 can be generated at least partially by parsing language model vocabulary 252. In at least one embodiment, the byte-level vocabulary representation vector set 254 can comprise at least two vectors, lists, or arrays. In at least one embodiment, the byte-level vocabulary representation vector set 254 can comprise a first vector whose elements contain byte representations of sorted tokens (sorted, for example, in increasing or decreasing order of numeric token identifiers assigned to tokens by a language model tokenizer, in alphabetical order, or in any other order) of language model vocabulary 252, and a second vector that stores starting indices of tokens within the first vector.In at least one embodiment, byte-level vocabulary representation vector set 254 can be generated using a parallel programming kernel, such as a CUDA kernel. In at least one embodiment, byte-level vocabulary representation vector set 254 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads to simultaneously generate byte-level representations of respective subsets of language model vocabulary 252. In at least one embodiment, byte-level vocabulary representation vector set 254 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads of a GPU or a machine learning accelerator to simultaneously generate byte-level representations of respective subsets of language model vocabulary 252.

[0019] In at least one embodiment, byte-level vocabulary mapping table 256 can be created based at least partially on analysis of byte-level vocabulary representation vector set 254 together with byte-level API input state representation 208. In at least one embodiment, byte-level vocabulary mapping table 256 can have a plurality of entries such that a given entry maps byte identifiers stored in a first vector of byte-level vocabulary representation vector set 264 to API input state transition identifiers displayed in byte-level API input state representation 208. In at least one embodiment, byte-level vocabulary mapping table 256 can be generated using a parallel programming kernel 233, such as a CUDA kernel.In at least one embodiment, byte-level vocabulary mapping table 256 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads to simultaneously generate entries that correspond to the respective subsets of a first vector of byte-level vocabulary representation vector set. In at least one embodiment, byte-level vocabulary mapping table 256 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads of a GPU or a machine learning accelerator to simultaneously generate entries that correspond to the respective subsets of a first vector of byte-level vocabulary representation vector set 254.

[0020] In at least one embodiment, language model output to API state mapping table 260 can be created based at least partially on analysis of byte-level vocabulary mapping table 256 together with API input state-to-state mapping table 210. In at least one embodiment, language model output to API state mapping table 260 can have a plurality of entries such that a given entry indicates, given a single current state of a state representation of accepted input tokens generated by one or more neural networks for a single API (e.g., a single current state of a byte-level FSM corresponding to a single API), whether the generation of a single token by the one or more neural networks will result in input that is unacceptable / invalid for the single API.In at least one embodiment, language model output to API state mapping table 260 can have a plurality of entries such that a given entry indicates, given a single current state of a state representation of accepted input tokens generated by one or more neural networks for a single API (e.g., a single current state of a byte-level FSM corresponding to a single API), whether the generation of a single token by the one or more neural networks will result in input that is acceptable / valid for the single API, e.g., whether the single token will lead to a valid next state. Thus, in at least one embodiment, language model output to API state mapping table can map tokens generated by one or more neural networks into tokens to be used by one or more APIs.In at least one embodiment, language model output to API state mapping table 260 can be generated using a parallel programming kernel 234, such as a CUDA kernel. In at least one embodiment, language model output to API state mapping table 260 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads to simultaneously generate entries corresponding to respective subsets of language model vocabulary 252, as shown in byte-level vocabulary mapping table 256. In at least one embodiment, language model output to API state mapping table 260 can be generated, at least partially in parallel, by assigning respective threads to a plurality of threads of a GPU or machine learning accelerator to simultaneously generate entries corresponding to respective subsets of language model vocabulary 252, as shown in byte-level vocabulary mapping table 256.

[0021] In at least one embodiment, during inference operations performed using one or more neural networks of a language model, language model output to API state mapping table 260 can be used to perform language model output token logits modification 267. In at least one embodiment, during inference operations performed using one or more neural networks of a language model, output token logits modification 267 can include setting logit values ​​generated at the one or more neural networks to negative infinity for one or more tokens (tokens that would cause invalid / unacceptable API input), thereby causing the probability of selecting these tokens for output to be set to zero at a given output generation stage.In at least one embodiment, during inference operations performed using one or more neural networks of a language model, output token logits modification 267 can include setting logit values ​​generated on the one or more neural networks to a large negative number for one or more tokens (tokens that would cause invalid / unacceptable API input), thereby causing the probability of selecting these tokens for output to be set to a very low value close to zero at a given level of output generation. In at least one embodiment, during inference operations performed using one or more neural networks of a language model, output token logits modification 267 can be performed using a parallel programming kernel 235, such as a CUDA kernel.In at least one embodiment, during inference operations performed using one or more neural networks of a language model, output token-logits modification 267 can be performed at least partially in parallel by assigning respective threads to a plurality of threads to examine and, if necessary, modify respective subsets of logits. In at least one embodiment, during inference operations performed using one or more neural networks of a language model, output token-logits modification 267 can be performed at least partially in parallel by assigning respective threads to a plurality of threads of a GPU or machine learning accelerator to examine and, if necessary, modify respective subsets of logits.

[0022] In at least one embodiment, at least a subset of tables and / or other data structures of the in Fig. 2. The type of API input state-to-state mapping table 210, byte-level vocabulary representation vector set 254, byte-level vocabulary mapping table 256, and / or language model output to API state mapping table 260 shown in Figure 2 can be pre-prepared for inference operations for a given language model and a given set of APIs. In at least one embodiment, at least a subset of tables and / or other data structures of the type shown in Figure 2 can be prepared for a given language model and a given set of APIs. Fig. The two types shown are stored in a cache at a language model agent execution environment and, where possible, reused to speed up the processing of end-user prompts (e.g., for different end-user prompts that may require calls to the same API or APIs).

[0023] Fig. Figure 3 illustrates an example of converting a regular expression indicating acceptable or valid API input into a representation of a finite state machine (FSM) according to at least one embodiment. In at least one embodiment, an FSM corresponding to a regular expression can be generated at least partially by API Constraint Compliance Manager 164. In at least one embodiment, acceptable or valid API input can be indicated by text sequences called regular expressions. For example, in at least one embodiment, API Acceptable Input Regular Expression 355 can contain a string “\d{3}-\d{3}-\d{4}”, indicating that acceptable input for an API consists of three digits followed by a dash, followed by three digits, followed by another dash, followed by four digits. In at least one embodiment, in the Fig. In the example shown, an API-acceptable regular input expression 355 represents or conforms to a format used for telephone numbers in the United States. In at least one embodiment, the Fig. The 3 examples shown are allowed tokens or alphabet characters for valid API input that correspond to API-acceptable-input-regular-expression 355, {0, 1, 2, 3, 5, 6, 7, 8, 9, -} (10 decimal digits and a dash symbol).

[0024] In at least one embodiment, the example of Fig. 3. An API-acceptable regular input expression 355 can be transformed into a finite state machine (FSM) 333 at the API token level, e.g., by API constraint compliance managers 164. In at least one embodiment, the API token-level FSM 333 can have a set of nodes and a set of edges linking pairs of nodes, where nodes represent states (such as start state S, state S0, state S9, end state E, etc.) and edges represent transitions between states (e.g., transition T0 corresponding to receiving a '0' as the next input, transition T9 corresponding to receiving a '9' as the next input, etc.). For the sake of simplicity, only some states and transitions are illustrated with rows of dots to show that the actual FSM would include many more states and transitions.

[0025] In at least one embodiment, the example of Fig. 3 API Token-Layer FSM 333 can be converted into an API Byte-Layer FSM 337, for example, by API Constraint Compliance Manager 164. In at least one embodiment, API Byte-Layer FSM 337 can have a set of nodes and a set of edges linking pairs of nodes, where nodes represent states corresponding to input bytes rather than whole tokens, and edges represent transitions between states resulting from the consumption or processing of additional bytes of input. In at least one embodiment, some tokens represented in API Token-Layer FSM 333, when converted to bytes, can consume or absorb multiple bytes, resulting in more total nodes and edges in API Byte-Layer FSM 337 than were present in API Token-Layer FSM 333.

[0026] Fig. Figure 4 illustrates an example of a workflow for mapping tokens of a neural network vocabulary to transition identifiers of an FSM representation of an acceptable or valid API input according to at least one embodiment. In at least one embodiment, at least one component of a Fig. The workflow shown in Figure 4 can be implemented by API Restriction Compliance Manager 164. In at least one embodiment, the language model vocabulary 405 of a language model that includes one or more neural networks can contain numerous token strings (such as complete words, partial words, numbers, phrases, etc.) to which individual token identifiers (token IDs) can be assigned, for example, by a tokenizer used for the one or more neural networks. In at least one embodiment, for example, tokens of a neural network vocabulary can include words such as "the," "and," and the line, as well as numbers such as "1," "2," or the like, and respective token IDs 1 for "the," 2 for "and," 3 for "1," and so on.

[0027] In at least one embodiment, corresponding to the respective bytes of allowed / valid input of an API, a respective transition identifier indicating a state transition of an API byte-level FSM 337 can be identified and stored, e.g., in a table or data structure similar to API input allowed bytes to transition ID. Fig. In at least one embodiment, for example, transition ID T0 can represent input b0 (a byte corresponding to the digit 0 is provided as input for an API-expecting input expressed according to regular expression 355), transition ID T1 can represent input b1 (a byte corresponding to the digit 1 is provided as input for an API-expected input expressed according to regular expression 355), transition ID T10 can represent input b- (a byte corresponding to the dash symbol “-” is provided as input for an API-expected input expressed according to regular expression 355), and so on.

[0028] In at least one embodiment, a byte-level vocabulary representation vector set 254, generated, for example, by API compliance constraint manager 164, can include a byte-level vocabulary token representation vector 460 and a vocabulary token start index vector 462. In at least one embodiment, the byte-level vocabulary token representation vector 460 can include a list of tokens (converted into constituent bytes) of language model vocabulary 405. In at least one embodiment, individual entries of the byte-level vocabulary token representation vector 460 can indicate or represent a specific byte of a specific vocabulary token.In at least one embodiment, individual entries of the byte-level vocabulary token representation vector 460 can be arranged according to a sorting criterion, such as in ascending order of token identifiers of the corresponding tokens or in descending order of token identifiers of the corresponding tokens. In at least one embodiment, start indices of individual tokens within the byte-level vocabulary token representation vector 462 can be displayed. For example, in at least one embodiment, an element or entry in the vocabulary token start index vector 462 can display a start index for bytes of "der / die / das" (e.g., an index where the byte identifier b0-die of the first byte of "der / die / das" within vector 460 is displayed), another element or entry in the vocabulary token start index vector 462 can display a start index for bytes of "und," and so on.

[0029] In at least one embodiment, transition IDs can be generated using at least API input-allowed bytes. Fig. , Byte-level vocabulary token representation vector 460 and vocabulary token start index vector 462, Byte-level vocabulary mapping table 256 can be constructed, e.g., by API constraint compliance manager 164. In at least one embodiment, one column of Byte-level vocabulary mapping table 256 can contain byte identifiers (vocab byte IDs) of byte-level representations of tokens of language model vocabulary 405, while another column contains transition identifiers (transition IDs) from API input-allowed bytes to transition ID Fig. can exhibit. In at least one embodiment, byte-level vocabulary mapping table 256 can be constructed using several parallel threads, e.g., where individual threads are assigned respective sections or portions of byte-level vocabulary token representation vector 460, for which corresponding transition identifiers are to be found and stored. In at least one embodiment, byte-level vocabulary mapping table 256 can be constructed by providing, for several parallel threads, respective sets of start indices, which are stored in vocabulary token start index vector 462, that indicate those tokens for which entries in byte-level vocabulary mapping table 256 are to be entered by individual threads.

[0030] Fig. Figure 5 illustrates an exemplary creation of a table that maps current API states to future API states based on bytes of API input, according to at least one embodiment. In at least one embodiment, API Byte-Level FSM 337 can be parsed to construct API Input State-to-State Mapping Table 210. In at least one embodiment, individual rows of a plurality of rows of API Input State-to-State Mapping Table 210 can be labeled with identifiers of respective states (API Input Current States 539) defined in API Byte-Level FSM 337. In at least one embodiment, individual columns of a plurality of columns of API input state-to-state mapping table 210 can be labelled with API byte-level transition IDs 537 (each corresponding to a respective byte received as input) displayed in API byte-level FSM 337.In at least one embodiment, API Input State-to-State Mapping Table 210 can be constructed at least partially by API Restriction Compliance Manager 164.

[0031] In at least one embodiment, a given entry of API input state-to-state mapping table 210, given a current state of API input and a next byte of input, can indicate the next state of API input. For example, in at least one embodiment, if the current state of an API input string being constructed is Sk and a byte corresponding to transition bj is received, API input state-to-state mapping table 210 can indicate that a next state of API input would be Sp.

[0032] Fig. Figure 6 illustrates an exemplary creation of a table that maps current API states to next API states based on tokens generated by a neural network, according to at least one embodiment. In at least one embodiment, language model output to API state mapping table 260 can be created at least partially by API constraint compliance manager 164. In at least one embodiment, language model output to API state mapping table 260 can be created, at least partially, by combining information contained in API input state-to-state mapping table 210 (which maps API input current states 539 to next states based on API byte-level transition IDs 537) and byte-level vocabulary mapping table 256 (which displays the respective state transition identifiers corresponding to the respective vocabulary byte identifiers).

[0033] In at least one embodiment, individual rows of a plurality of rows from language model output to API state mapping table 260 can represent respective API input current states 539. In at least one embodiment, individual API input current states 539 can correspond to any sequence of accepted API input tokens; as such, language model output to API state mapping table 260 can map tokens generated by one or more neural networks of a language model to one or more tokens to be used by an API. In at least one embodiment, individual columns of a plurality of columns from language model output to API state mapping table 260 can represent respective vocabulary tokens 637.In at least one embodiment, individual columns of a plurality of columns from language model output to API state mapping table 260 can represent respective individual bytes of vocabulary tokens 637. In at least one embodiment, a given entry from language model output to API state mapping table 260 can indicate, given a single current state of a state representation of accepted input tokens generated by one or more neural networks for a single API, whether the generation of a single token by one or more neural networks of the language model will result in input that is not acceptable for the single API.In at least one embodiment, to indicate that generating a single token of a neural network vocabulary will result in an unacceptable or invalid input for an API, a special value, such as "-1", can be stored in a corresponding entry in Language Model Output to API State Mapping Table 260. In at least one embodiment, a given entry of Language Model Output to API State Mapping Table 260 can indicate, given a single current state of a state representation of accepted input tokens generated by one or more neural networks for a single API, whether generating a single token by the one or more neural networks will result in input that is acceptable for the single API.In at least one embodiment, to indicate that generating a single token of a neural network vocabulary will result in acceptable or valid input for an API, an identifier of the next state that would be reached as a result of the single token can be stored in the language model output to API state mapping table 260. For example, in at least one embodiment, an entry with coordinates Sg and Tok-q in the language model output to API state mapping table can store state identifier Sn (if Sn is a valid next state reached by consuming token Tok-q as input when the current state is Sg) or -1 (if token Tok-q would necessarily result in invalid or unacceptable API input).In at least one embodiment, to populate a single entry in language model output to API state mapping table 260, a logical walk can be performed within API input state-to-state mapping table 210, wherein individual steps of the logical walk are performed based on respective byte-level transition identifiers that correspond to successive bytes of a language model output token, until a valid next state of API input is reached or a determination is made that no valid next state can be reached.

[0034] In at least one embodiment, language model output to API state mapping table 260 can be constructed using multiple parallel threads, e.g., where individual threads are assigned respective subsets or portions of a language model vocabulary for which the threads are to generate entries of language model output to API state mapping table 260. In at least one embodiment, respective sections or portions of language model output to API state mapping table 260 can be constructed using multiple parallel threads of a GPU or a machine learning accelerator.

[0035] Fig. Figure 7 illustrates an exemplary inference-time modification, using one or more mapping tables, of output probabilities assigned to tokens on a neural network, according to at least one embodiment. In at least one embodiment, initial logit values ​​762, produced during inference operations to generate a single next token as output by one or more neural networks of a language model, can have a respective numerical value (e.g., 0.03, 0.59, 1.3, or 0.0006) for each of at least one subset of tokens (e.g., tokens with respective token IDs 1, 2, 3, and 20000) of a vocabulary of the neural network.In at least one embodiment, respective logit values ​​produced for different tokens of a vocabulary during inference operations to generate a single next token as output by one or more neural networks of a language model can be converted within the one or more neural networks into an output probability (such that a sum of the output probabilities is one), e.g., using one or more mathematical functions.

[0036] In at least one embodiment, language model output to API state mapping table 260 can be examined while inference operations of one or more neural networks of a language model are performed to identify those tokens whose generation would result in invalid API input. In at least one embodiment, using entries that indicate invalid next states (e.g., expressed using -1) in language model output to API state mapping table 260, corresponding logit entries can be set to negative infinity in modified logit values ​​763, such that the probability of those tokens being generated is zero.In at least one embodiment, for example, if the current state of API input (a state corresponding to a set of tokens already generated by a neural network for calling an API, where the set of tokens generated so far represents acceptable or valid input) is Sg, and generating the next token Tok-q would result in invalid API input, a logit value corresponding to Tok-q can be set to negative infinity. In at least one embodiment, values ​​other than negative infinity (such as large finite negative numbers) can be used. In at least one embodiment, by setting logit values ​​for tokens whose generation would reasonably result in invalid API input, only those tokens whose generation could still result in valid API input can be assigned non-zero output probabilities.In at least one embodiment, instead of modifying logit values, as in . Fig. Figure 7 shows that output probabilities generated after converting logits to output probabilities can be directly modified to reduce or eliminate the generation of tokens that would result in invalid API input.

[0037] In at least one embodiment, modification of logit values ​​can be performed using a plurality of parallel threads. In at least one embodiment, modification of logit values ​​can be performed using a plurality of parallel threads of a GPU or a machine learning accelerator. In at least one embodiment, respective threads can be assigned to a plurality of threads to examine language model output to API state mapping table 260 for respective subsets of a vocabulary and to modify logit values ​​corresponding to the respective subsets of the vocabulary.

[0038] Fig. Figure 8 illustrates a method for generating and using mapping tables for calling LLM agent APIs according to at least one embodiment. In at least one embodiment, the Fig. The procedures illustrated in section 8 can be implemented at least partially by an API Restriction Compliance Manager 164, which is responsible for Fig. 1 is discussed, and / or various different embodiments of systems, applications, services or devices, which are discussed below in relation to Fig. 10 to 28C are discussed. In at least one embodiment, as shown in 802, a display of one or more constraints on acceptable input can be received from one or more APIs that can be called by a language model having one or more neural networks to respond to an input prompt, for example, via one or more programmatic interfaces of a language model service or system. In at least one embodiment, a display of one or more constraints on acceptable input can be called from one or more APIs that can be called by a language model or associated agent as part of an input prompt to the language model for which the one or more APIs can be called.In at least one embodiment, a display of one or more restrictions on acceptable input from one or more APIs that can be called by a language model may include, for example, one or more regular expressions and / or JSON files, schemas, descriptors or objects.

[0039] In at least one embodiment, a display of vocabulary tokens of a language model can be obtained, as shown in 804. In at least one embodiment, a list of tokens that a language model can produce as output can be obtained via one or more tools provided by a vendor or developer of a language model or by other parties.

[0040] In at least one embodiment, a state machine representation (e.g., an FSM) of acceptable byte-level input tokens of an API can be constructed from constraints of acceptable input, as shown in 806. In at least one embodiment, a state machine representation of acceptable byte-level input of an API can be constructed in at least two stages—a first stage in which a first state machine representation is constructed at a token or alphabet character level, and a second stage in which a second state machine representation is generated by extending or replacing tokens in the first state machine representation with respective constituent bytes of tokens in scenarios where at least some tokens occupy multiple bytes.In at least one embodiment, as also shown in 806, a list of state transition identifiers can be constructed that correspond to different bytes of API input.

[0041] In at least one embodiment, a vector set VS can be generated, comprising a vector V1 containing byte-level representations of tokens of a language model vocabulary, and vector V2 providing indices of starts of different tokens within V1, as shown in 808. In at least one embodiment, vector set VS can be generated using a plurality of threads that operate at least partially in parallel.

[0042] In at least one embodiment, for example, based on parsing an FSM of acceptable byte-level tokens of API input, an API input state-to-state mapping table can be generated that indicates the next state of an API input string, given a current state and a next byte, as shown in 810. In at least one embodiment, an API input state-to-state mapping table can map a current state of API input and a next API input byte to a next state of API input.

[0043] In at least one embodiment, as shown in 812, a byte-level vocabulary mapping table can be generated using a list of transition IDs generated in 806 and one or more vectors of a vector set generated in 808. In at least one embodiment, a byte-level vocabulary mapping table can be generated using a plurality of threads that operate at least partially in parallel. In at least one embodiment, a byte-level vocabulary mapping table can map bytes or byte identifiers of vocabulary tokens represented in a vector set generated in 808 to API input state transition identifiers represented in a list of transition IDs generated in 806.

[0044] In at least one embodiment, as shown in 820, a language model output to API state mapping table can be created that maps current API input states and vocabulary tokens to the next API states (or invalid states for tokens that would not be acceptable for an API). In at least one embodiment, a language model output to API state mapping table can be created at least partially by parsing a combination of an API input state-to-state mapping table and a byte-level vocabulary mapping table. In at least one embodiment, a language model output to API state mapping table can be created using a plurality of threads that operate at least partially in parallel, for example, with individual threads to which respective subsets of a vocabulary of the language model are assigned.

[0045] In at least one embodiment, as shown in 830, during inference operations performed on a language model comprising one or more neural networks, language model output to API state mapping table, generated in 820, can be used to modify logit values ​​assigned to one or more tokens of a vocabulary of the language model. For example, in at least one embodiment, logit values ​​for the next tokens that would result in unacceptable API input can be set to negative infinity, thus setting the probabilities of selecting these tokens as output at a given output generation step to zero, thereby ensuring that API input constraints are met.In at least one embodiment, logits can be modified using multiple threads that operate at least partially in parallel to each other, for example with individual threads to which respective subsets of the logits are assigned.

[0046] Fig. Figure 9 illustrates an exemplary system in which mapping tables for APIs can be dynamically generated and cached for reuse with resources of an inference service, according to at least one embodiment. In at least one embodiment, clients of LLM-Inference-as-a-Service 904 can be provided with access to one or more Large Language Models (LLMs) of LLM-Collection 906. In at least one embodiment, individual LLMs of LLM-Collection 906 can be selected by clients of LLM-Inference-as-a-Service 904 to generate responses to input prompts. In at least one embodiment, at least some LLMs of LLM-Collection 906 can include one or more neural networks that can generate output for calling or invoking APIs from one or more external tools.

[0047] In at least one embodiment, inference requests that have input prompts from several different categories can be received via one or more programmatic interfaces of LLM Inference-as-a-Service 904. In at least one embodiment, to prepare responses to inference requests that have input prompts from one or more categories, respective sets of one or more APIs can be called by (or called by an LLM agent using output generated by) one or more LLMs to which the inference requests are directed.In at least one embodiment, for example, Category C1 inference requests may require calling APIs from API set AS1 of external toolset TS1, Category C2 inference requests may require calling APIs from API set AS2 of external toolset TS2, and Category C3 inference requests may require calling APIs from API set AS3 of external toolset TS3. In at least one embodiment, at least some APIs and / or corresponding API input constraints may be indicated within the inference requests for which these APIs may need to be called. In at least one embodiment, the set of APIs that a given LLM or LLM agent of LLM Inference-as-a-Service must call may grow over time as inference requests of more categories are received.

[0048] In at least one embodiment, LLM-Inference-as-a-Service 904 can include one or more API Restriction Compliance Managers 164 that generate mapping tables and other data structures similar to those used with respect to Fig. 2 - Fig. 8 above are discussed to ensure compliance of output generated by LLMs with input constraints of APIs. In at least one embodiment, for example, mapping table(s) 920A and / or other data structures can be generated for APIs required to respond to inference requests of category C1, mapping table(s) 920B and / or other data structures can be generated for APIs required to respond to inference requests of category C2, mapping table(s) 920C and / or other data structures can be generated for APIs required to respond to inference requests of category C3, and so on. In at least one embodiment, a single inference request can result in calls to multiple APIs.

[0049] In at least one embodiment, after mapping tables and / or other data structures have been generated for a given LLM and a given set of APIs, the mapping tables and / or other data structures can be stored in mapping table caches 922 for potential reuse when processing additional inference requests. In at least one embodiment, for example, mapping tables and / or other data structures useful for modifying logits, as in Fig. Figure 7 shows that inference operations are stored in caches accessible by GPUs or other machine learning accelerators, which perform inference operations in response to inference requests. In at least one embodiment, when a mapping table useful for modifying logits or output probabilities is moved or removed from a cache, for example, due to insufficient space as more and more such tables are created and cached, it can be relocated using techniques described above with respect to Fig. 1 - Fig. 8 will be discussed, efficiently regenerated. DATA CENTER

[0050] Fig. Figure 10 illustrates an exemplary data center 1000 in which at least one embodiment can be used. The data center 1000 can include one or more rooms containing racks 1002 and auxiliary equipment used to house one or more racks 1002 and one or more base plates 1004. A rack 1002 can contain one or more base plates 1004. A rack 1002 can include an enclosure that accommodates and supports individual base plates 1004. Operational aspects of the rack 1002 can be regulated, among other options, at a rack level corresponding to a group of base plates 1004 or at a base plate level corresponding to individual base plates 1004. In particular, the rack 1002 or base plates 1004 can have selected maximum operating parameters, such as, but not limited to, power consumption, operating frequencies, and others.Data center 1000 can be supported by various cooling systems, such as, but not limited to, cooling towers, cooling circuits, pumps, and other support systems. Cooling systems can include sensors and controllers for monitoring and managing the cooling characteristics of racks 1002. Base plates 1004 within racks 1002 can receive power from one or more power distribution units (PDUs; not shown). PDUs can be located within racks 1002, for example, between racks 1002 containing base plates 1004, or within racks 1002 that also house base plates 1004.

[0051] Racks 1002 and base plates 1004 can contain subsystems, modules, add-in cards, and other semiconductor components. Base plates 1004 can contain one or more computing units 1006, which can contain one or more processors 1008, one or more memory units 1010, and an interface controller 1012. Computing units 1006 can contain any number of processors, such as, but not limited to, central processing units (“CPUs”), graphics processing units (“GPUs”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), including any processors described herein, such as, but not limited to, the processors in Fig. 11- Fig. 23. Computing Units 1006 can include one or more Storage Devices 1010 (e.g., dynamic read-only memory, solid-state storage, or disk drives), as well as Network Input / Output Devices (“NW I / O”), Network Switches, Virtual Machines (“VMs”), Power Modules, and Cooling Modules, etc. One or more Computing Units 1006 can constitute a server with one or more of the above-mentioned computing resources.

[0052] Computing units 1006 can include separate groupings of compute units located in one or more racks (not shown), or many racks located in data centers at different geographic locations (also not shown). Separate groupings of compute units can include grouped compute, network, short-term storage, or long-term storage resources that can be configured or allocated to support one or more workloads. Multiple compute units (e.g., including CPUs and / or other processors) can be grouped within one or more racks to provide compute resources to support one or more workloads. A resource orchestrator 1014 can configure or otherwise control one or more compute units 1006 or groups of compute units.Resource Orchestrator 1014 can include a Software Design Infrastructure (“SDI”) management entity for Data Center 1000. Resource Orchestrator 1014 can include hardware, software, or a combination thereof.

[0053] Data Center 1000 can contain any one or any combination of a framework layer 1020, a software layer 1030, and an application layer 10340. As in Fig. As shown in Figure 10, framework layer 1020 includes a job scheduler 1022, a configuration manager 1024, a resource manager 1026, and a distributed file system 1028. Framework layer 1020 can include a framework to support software 1032 from software layer 1030 and / or one or more applications 1042 from application layer 1040. Software 1032 or application(s) 1042 can each include, but are not limited to, web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 can be a type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which can use distributed file system 1028 for large-scale data processing (e.g., "Big Data").Job Scheduler 1022 can include a Spark driver to facilitate scheduling workloads supported by various layers of Data Center 1000. Configuration Manager 1024 can configure various layers, such as, but not limited to, Software Layer 1030 and Framework Layer 1020, including Spark and Distributed File System 1028, to support large-scale computing. Resource Manager 1026 can manage clustered or grouped compute units 1006, mapped to or allocated to support Distributed File System 1028 and Job Scheduler 1022. Resource Manager 1026 can coordinate with Resource Orchestrator 1014 to manage these mapped or allocated compute resources.

[0054] Software 1032 can be contained in software layer 1030 and can include software used by at least portions of a computing unit 1006, one or more computing units 1006, groups of computing units 1006, and / or a distributed file system 1028 of framework layer 1020. One or more types of software can include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0055] Application(s) 1042 may be contained in application layer 1040 and may include one or more types of applications used by at least portions of a compute unit 1006, one or more compute units 1006, groups of compute units 1006, and / or distributed file system 1028 of framework layer 1020. One or more types of applications may, but are not limited to, include any number of a genomic application, a cognitive computing application, and a machine learning application, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0056] Each of the Configuration Manager 1024, Resource Manager 1026, and Resource Orchestrator 1014 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible way. Self-modifying actions can relieve a Data Center 1000 operator of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing parts of a data center.

[0057] Data Center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to predict or derive information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and computing resources described above in relation to Data Center 1000. Trained machine learning models corresponding to one or more neural networks may be used to derive or predict information using resources described above in relation to Data Center 1000 by using weight parameters calculated by one or more training techniques described herein.

[0058] Data Center 1000 can accommodate CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., embodiments in Fig. 11- Fig. 23) to perform some or all of the processes and techniques described elsewhere herein, such as, but not limited to, training and / or inference using the resources described above. In addition, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or perform information inferences, such as image recognition, speech recognition, or other artificial intelligence services.

[0059] In at least one embodiment, processor 1008 may include one of the processors listed below and / or have one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.In at least one embodiment, processor 1008 is configured by software 1032 to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens that are to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform one of the operations described above or elsewhere herein. Data center 1000 can include logic, CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., embodiments in [reference]). Fig. 11- Fig. 23) use to perform any of the operations described above or elsewhere herein. PROCESSORS

[0060] The following figures represent, without limitation, exemplary processors and processing systems that can be used to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise perform some or all of the processes, operations and / or techniques described elsewhere herein.Exemplary processors and processing systems can be configured by software to cause one or more tables to map one or more tokens, generated by one or more neural networks, into one or more tokens that can be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. Processors and processing systems can be logic, central processing units (CPUs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), field-programmable arrays (FPGAs), XPUs (i.e., any computing architecture best suited to the needs of an application), or other hardware (e.g., embodiments in [reference]). Fig. 11- Fig. 23) include, to perform any of the operations described above, below, or elsewhere herein. Processors and / or processing systems described herein may include one or more circuits that can be used to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.As used herein, one or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens that can be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein. Fig. 28A and Fig. Figure 28B illustrates logic 2815, which, as described elsewhere herein, can be used in one or more devices to perform operations such as, but not limited to, those discussed herein according to at least one embodiment. Logic may, for example, refer to any combination of software logic, hardware logic, and / or firmware logic to provide functionality and / or operations described herein, the logic, collectively or individually, being embodied as a circuit that forms part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a field-programmable array (FPGA), a system-on-a-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0061] Fig. Figure 11 illustrates a processor that is a System-on-a-Chip (SOC) 1100 (which may be referred to as a System-on-Chip, a Superchip, or by another name) according to at least one embodiment. SOC 1100 may include processor complex 1110 and processor complex 1140. SOC 1100 may include any number of processor complexes 1110 and / or processor complexes 1140, which may include any number of processors described herein, such as, but not limited to, those in Fig. 11- Fig. 23, in any combination. For example, processor 1110 may contain a central processing unit (CPU) and processor 1140 may contain a graphics processing unit (GPU). Alternatively, processor 1110 may contain a GPU and processor 1140 may contain a GPU. SOC 1100 may contain any number of display controllers 1192, any number of multimedia engines 1194, any number of I / O interfaces 1170, any number of memory controllers 1180, and any number of fabrics 1160 in any combination. For explanatory purposes, multiple instances of similar objects are referred to herein by reference numbers that identify the object and, where needed, by bracket numbers that identify the instance. SOC 1100 may contain a processor made by Broadcom in Palo Alto, CA.

[0062] Processor complex 1110 can include a CPU, processor complex 1140 can include a GPU, and SOC 1100 can be a processing unit that integrates the 1110 and 1140 on a single chip. Some tasks can be assigned to processor complex 1110, and other tasks can be assigned to processor complex 1140. Processor complex 1110 can be configured to run main control software associated with SOC 1100, such as, but not limited to, an operating system. Processor complex 1110 can be the main processor of SOC 1100, controlling and coordinating operations of other processors. Processor complex 1110 can issue instructions that control the operation of processor complex 1140 to perform some or all of the operations described herein. Processor complex 1110 can be configured to execute host executable code provided by CUDA or other source code (e.g.,HIP source code) is derived, and processor complex 1140 can be configured to execute device executable code derived from CUDA or other source code to perform any of the operations described herein.

[0063] Processor complex 1110 can include cores 1120(1)-1120(4) and a cache (e.g., L3 cache) 1130 to store information for performing the operations described herein. Processor complex 1110 can include any number of cores 1120 and any number and type of caches in any combination. Cores 1120 can be configured to execute instructions of a single instruction set architecture (“ISA”) to perform some or all of the operations described herein. Each core 1120 can contain one CPU core. Cores 1120(1)-1120(4) can be referred to as an arithmetic unit or arithmetic units. SOC 1100 can include any number of processor complexes 1110, fabric 1160, I / O interfaces 1170, and memory controllers 1180.

[0064] Each 1120 core can include a fetch / decode unit 1122, an integer execution machine 1124, a floating-point execution machine 1126, and an L2 cache 1128. The fetch / decode unit 1122 can fetch instructions to perform some or all of the operations described herein (such as, but not limited to, an API compiled into instructions) and decode such instructions, generate microoperations, and send separate microinstructions to the integer execution machine 1124 and / or the floating-point execution machine 1126. The fetch / decode unit 1122 can simultaneously send one microinstruction to the integer execution machine 1124 and another microinstruction to the floating-point execution machine 1126. The integer execution machine 1124 can perform integer and memory operations. The 1126 floating-point machine can perform floating-point and vector operations.The retrieval / decoding unit 1122 can send micro-instructions to one or more execution machines, replacing both the integer execution machine 1124 and the floating-point execution machine 1126.

[0065] Each core 1120(i), where i is an integer representing a single instance of core 1120, can access L2 cache 1128(i), contained within core 1120(i). Each core 1120, contained within core complex 1110(j), where j is an integer representing a single instance of core complex 1110, can be connected to other cores 1120, contained within core complex 1110(j), via L3 cache 1130(j), contained within core complex 1110(j). Cores 1120, contained within core complex 1110(j), where j is an integer representing a single instance of core complex 1110, can access all L3 caches 1130(j), contained within core complex 1110(j). L3 cache 1130 can contain any number of slices.

[0066] Processor Complex 1140 can be a graphics complex that can be configured to perform computational operations (e.g., computational operations involved in the operations described herein) in a highly parallel manner. Processor Complex 1140 can be configured to perform graphics pipeline operations, such as, but not limited to, drawing commands, pixel operations, geometric calculations, and other operations associated with rendering an image on a display. Processor Complex 1140 can be configured to perform operations that are not related to graphics, such as, but not limited to, training and / or simulating neural networks. Processor Complex 1140 can be configured to perform both graphics-related and non-graphics-related operations.

[0067] A processor complex 1140 can include any number of arithmetic units 1150(1)-1150(N), where N is an integer greater than 1, and an L2 cache 1142. Arithmetic units 1150 can share the L2 cache 1142, which can store information to be used to perform some or all of the operations described herein. The L2 cache 1142 can be partitioned. A processor complex 1140 can include any number of arithmetic units 1150 and any number (including zero) and type of caches. A processor complex 1140 can include any amount of dedicated graphics hardware.

[0068] Each compute unit 1150 can contain any number of SIMD units 1152(1)-1152(N), where N is an integer greater than 1, and a shared memory 1154. Each SIMD unit 1152 can implement a SIMD architecture and can be configured in parallel for some or all of the operations described herein. Each compute unit 1150 can execute any number of thread blocks, but each thread block can be executed on a single compute unit 1150, although in some embodiments a thread block can be executed on multiple compute units. A thread block can contain any number of execution threads. A workgroup can be a thread block. Each SIMD unit 1152 can execute a group of threads. A group of threads (e.g., 16 threads), which can also be called a warp, subgroup, or wavefront (e.g.,(as used by AMD and Intel), where each thread in the warp, wave, subgroup, or wavefront can belong to a single thread block and is configured to process a different data set based on a single set of instructions. Prediction can be used to disable one or more threads in a warp, subgroup, or wavefront. A lane can be a thread. A worker can be a thread, as with OpenCL, but is not limited to that. Different warps, subgroups, or wavefronts in a thread block can synchronize with each other and communicate via shared memory. Each compute unit can contain one or more thread block clusters, where a thread block cluster can allow programmatic control of locality at a granularity greater than that of a single thread block of a single streaming multiprocessor (SM).Thread block clusters (also referred to as "clusters") enable multiple thread blocks running concurrently on streaming multiprocessors to synchronize and collaboratively retrieve, exchange, or otherwise use data. In at least one embodiment, streaming multiprocessors ("SMs") can be referred to as streaming microprocessors, stream processors ("SPs"), stream processing units ("SPUs"), compute units ("CUs"), execution units ("EUs"), and / or slices, where a slice in this context can refer to a portion of processing resources within a processing unit (e.g., 16 cores, a ray tracing unit, a thread director, or a scheduler).

[0069] Fabric 1160 can be a system assembly that facilitates data and control transfers across Processor Complex 1110, Processor Complex 1140, I / O Interfaces 1170, Memory Controller 1180, Display Controller 1192, and Multimedia Engine 1194, for example, to perform some or all of the operations described herein. SOC 1100 can include any number and type of system assembly in addition to or instead of Fabric 1160, facilitating data and control transfers across any number and type of directly or indirectly linked components, which may be located inside or outside of SOC 1100. I / O Interfaces 1170 can represent any number and type of I / O interface (e.g., PCI, PCI-Extended (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices can be connected to I / O interfaces 1170.Peripheral devices that can be coupled with I / O interfaces 1170 can include keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so on.

[0070] The display controller 1192 can display images on one or more display devices, such as, but not limited to, a liquid crystal display (“LCD”) device. The multimedia engine 1194 can include any quantity and type of switching technology related to multimedia, such as, but not limited to, a video decoder, a video encoder, an image signal processor, etc. The memory controller 1180 can facilitate data transfers between the SOC 1100 and a unified system memory 1190. The processor complex 1110 and processor complex 1140 can jointly utilize the unified system memory 1190. The unified system memory 1190 can include various types of memory devices, including dynamic random-access memory (DRAM) or graphics random-access memory, such as, but not limited to, synchronous graphics random-access memory (SGRAM), including graphics double-rate data rate (GDDR) memory.Unified System Memory 1190 can include 3D stacked memory, including but not limited to High Bandwidth Memory (HBM), HBM2e or HDM3.

[0071] SOC 1100 can implement a memory subsystem that includes any number and type of memory controllers 1180 and memory devices (e.g., shared memory 1154) that can be dedicated to a component or shared by multiple components to perform any of the operations described herein. SOC 1100 can implement a cache subsystem that includes one or more cache memories (e.g., L2 caches 1128, L3 cache 1130, and L2 cache 1142) that can each be private or shared by any number of components (e.g., cores 1120, core complex 1110, SIMD units 1152, compute units 1150, and processor complex 1140).

[0072] In at least one embodiment, SOC 1100 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein.

[0073] Fig. Figure 12A illustrates a parallel processor 1200 according to at least one embodiment. Parallel processor 1200 can be implemented using one or more circuits and can be a programmable processor (e.g., a CPU and / or GPU), logic, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other hardware (e.g., embodiments in Fig. 11- Fig. 23) are designated to perform any of the operations described above or elsewhere herein.

[0074] The Parallel Processor 1200 can include a Parallel Processing Unit 1202 to perform any of the operations described above or elsewhere herein. The Parallel Processing Unit 1202 can include an I / O Unit 1204, which enables communication with other devices, including other instances of the Parallel Processing Unit 1202. The I / O Unit 1204 can be directly connected to other devices. The I / O Unit 1204 can connect to other devices using a hub or switch interface, such as, but not limited to, a Memory Hub 1205. Connections between the Memory Hub 1205 and the I / O Unit 1204 can form a Communication Link 1213.The I / O unit 1204 can connect to a host interface 1206 and a memory crossbar 1216, with the host interface 1206 receiving commands aimed at performing processing operations and the memory crossbar 1216 receiving commands aimed at performing memory operations.

[0075] When host interface 1206 receives a command buffer via I / O unit 1204, host interface 1206 can route work operations to execute these commands to a front end 1208. Front end 1208 can couple with a scheduler 1210 (which can be described as a sequencer) configured to distribute commands or other work items to a processing cluster array 1212. Scheduler 1210 can ensure that processing cluster array 1212 is properly configured and in a valid state before tasks can be distributed to a cluster of processing cluster array 1212. Scheduler 1210 can be implemented via firmware logic running on a microcontroller.The microcontroller-implemented Scheduler 1210 can be configured to perform complex scheduling and workload distribution operations with coarse and fine granularity, enabling fast preemption and context switching of threads running on Processing Array 1212. Host software can allocate workloads for scheduling on Processing Cluster Array 1212 via one of several graphics processing paths. Workloads can then be automatically distributed across Processing Cluster Array 1212 by Scheduler 1210 logic within a microcontroller that incorporates Scheduler 1210.

[0076] Processing cluster array 1212 can perform any of the operations described above or elsewhere herein and can contain up to "N" processing clusters (e.g., cluster 125A, cluster 125B, up to cluster 125N), where "N" is a positive integer (which may be a different integer "N" than used in other figures). Each cluster 125A-125N of processing cluster array 1212 can execute a large number of concurrent threads. Scheduler 1210 can allocate work to clusters 1214A-1214N of processing cluster array 1212 using various scheduling and / or workload distribution algorithms, which may vary depending on the workload generated by each type of program or computation. Planning can be handled dynamically by planner 1210 or can be partially supported by compiler logic during compilation of program logic configured for execution by processing cluster array 1212.Different clusters 1214A -1214N of processing cluster array 1212 can be assigned to process different types of programs or to perform different types of calculations.

[0077] Processing Cluster Array 1212 can be configured to perform various types of parallel processing operations, such as any of the operations described above or elsewhere herein. Processing Cluster Array 1212 can be configured to perform general-purpose parallel computing operations. For example, Processing Cluster Array 1212 can include logic to perform processing tasks, including filtering video and / or audio data, performing modeling operations, including physics operations, and performing data transformations.

[0078] The processing cluster array 1212 can be configured to perform parallel graphics processing operations. The processing cluster array 1212 can include additional logic to support the execution of such graphics processing operations, including, but not limited to, texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. The processing cluster array 1212 can be configured to execute graphics processing-related shader programs, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. The parallel processing unit 1202 can transfer data from system memory via the I / O unit 1204 for processing. During processing, transferred data can be stored in on-chip memory (e.g., parallel processor memory 1222) and then written back to system memory.

[0079] When parallel processing unit 1202 is used to perform graphics processing, scheduler 1210 can be configured to divide a processing workload into approximately equal-sized tasks to better distribute graphics processing operations across multiple clusters 1214A–1214N of processing cluster array 1212. Portions of processing cluster array 1212 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display.Intermediate data generated by one or more of the clusters 1214A-1214N can be stored in buffers to allow intermediate data to be transferred between clusters 1214A-1214N for further processing.

[0080] Processing cluster array 1212 can receive processing tasks to be executed by scheduler 1210, which receives commands defining processing tasks from frontend 1208. Processing tasks can include indices of data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands that define how data is to be processed (e.g., which program to execute). Scheduler 1210 can be configured to retrieve indices corresponding to tasks or can receive indices from frontend 1208. Frontend 1208 can be configured to ensure that processing cluster array 1212 is configured in a valid state before initiating a workload specified by incoming command buffers (e.g., batch buffers, push buffers, etc.).

[0081] Each of one or more instances of a parallel processing unit 1202 can couple to parallel processor memory 1222 to perform any of the operations described above or elsewhere herein. Parallel processor memory 1222 can be accessed via memory crossbar 1216, which can receive memory requests from processing cluster array 1212 and I / O unit 1204. Memory crossbar 1216 can access parallel processor memory 1222 via a memory interface 1218. Memory interface 1218 can contain multiple partitioning units (e.g., partitioning unit 1220A, partitioning unit 1220B, up to partitioning unit 1220N), each of which can couple to a portion (e.g., storage unit) of parallel processor memory 1222.A number of partitioning units 1220A–1220N can be configured to be equal to a number of storage units, such that a first partitioning unit 1220A has a corresponding first storage unit 1224A, a second partitioning unit 1220B has a corresponding storage unit 1224B, and an Nth partitioning unit 1220N has a corresponding Nth storage unit 1224N. A number of partitioning units 1220A–1220N can be different from a number of storage units.

[0082] Memory units 1224A–1224N can contain various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Memory units 1224A–1224N can also contain 3D stacked memory, including, but not limited to, high-bandwidth memory (HBM), HBM2e, or HDM3. Render targets, such as, but not limited to, frame buffers or texture maps, can be stored across memory units 1224A–1224N, allowing partitioning units 1220A–1220N to write portions of each render target in parallel to efficiently utilize available bandwidth of parallel processor memory 1222.A local instance of parallel processor memory 1222 can be excluded in favor of a unified memory design that uses system memory in conjunction with local cache memory.

[0083] Each of the clusters 1214A-1214N of processing cluster array 1212 can process data written to any of the storage units 1224A-1224N within parallel processor memory 1222. Memory crossbar 1216 can be configured to transfer an output from each cluster 1214A-1214N to any partitioning unit 1220A-1220N or to another cluster 1214A-1214N, which can perform additional processing operations on the output. Each cluster 1214A-1214N can communicate via memory interface 1218 through memory crossbar 1216 to read from or write to various external storage devices.Memory crossbar 1216 can have a connection to memory interface 1218 to communicate with I / O unit 1204, as well as a connection to a local instance of parallel processor memory 1222, which allows processing units within different processing clusters 1214A–1214N to communicate with system memory or other memory that is not local to parallel processing unit 1202. Memory crossbar 1216 can use virtual channels to separate traffic flows between clusters 1214A–1214N and partitioning units 1220A–1220N.

[0084] Multiple instances of Parallel Processing Unit 1202 can be deployed on a single add-in card, or multiple add-in cards can be interconnected. Different instances of Parallel Processing Unit 1202 can be configured to work together compatiblely, even if they have different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, some instances of Parallel Processing Unit 1202 may include higher-precision floating-point units relative to other instances.Systems incorporating one or more instances of Parallel Processing Unit 1202 or Parallel Processor 1200 can be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop or handheld personal computers, servers, workstations, game consoles and / or embedded systems.

[0085] Fig. 12A further includes a block diagram of a partitioning unit 1220 according to at least one embodiment. Partitioning unit 1220 is an instance of one of the partitioning units 1220A to 1220N of Fig. 12A. Partitioning unit 1220 can include an L2 cache 1221, a frame buffer interface 1225, and a ROP 1226 (raster operations unit). L2 cache 1221 can be a read / write cache configured to perform load and store operations received from memory crossbar 1216 and ROP 1226. Read failures and urgent write-back requests can be passed from L2 cache 1221 to frame buffer interface 1225 for processing. Updates can also be sent to a frame buffer via frame buffer interface 1225 for processing. Frame buffer interface 1225 can be linked to any of the memory units in parallel processor memory, such as, but not limited to, memory units 1224A–1224N. Fig. 12A (e.g., within parallel processor memory 1222).

[0086] ROP 1226 can be a processing unit that performs raster operations, such as, but not limited to, stenciling, z-testing, blending, etc. ROP 1226 can then output processed graphics data stored in graphics memory. ROP 1226 can include compression logic to compress depth or color data written to memory and to decompress depth or color data read from memory. Compression logic can be lossless, employing one or more of several compression algorithms. The type of compression performed by ROP 1226 can vary based on the statistical characteristics of the data being compressed. For example, delta color compression is performed on depth and color data on a per-tile basis.

[0087] ROP 1226 can be used within any processing cluster (e.g., clusters 1214A - 1214N of Fig. 12A) instead of being contained within partitioning unit 1220. Read and write requests for pixel data can be submitted via memory crossbar 1216 instead of pixel fragment data. Processed graphics data can be displayed on a screen, which is then routed to processor(s) 2002 for further processing, or to one of the processing entities within parallel processor 1200. Fig. 12A will be routed.

[0088] In at least one embodiment, the parallel processor 1200 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein.

[0089] Fig. Figure 12B includes a block diagram of a processing cluster 1214 within a parallel processing unit according to at least one embodiment. A processing cluster can be an instance of one of the processing clusters 1214A to 1214N of Fig. Processing cluster 12A can be used to perform any of the operations described above or elsewhere herein. Processing cluster 1214 can be configured to run many threads in parallel, where "thread" refers to an instance of a particular program running on a specific set of input data. Single-instruction multiple data (SIMD) instruction output techniques can be used to support the parallel execution of a large number of threads without providing multiple independent instruction units. Single-instruction multiple thread (SIMT) techniques can be used to support the parallel execution of a large number of generally synchronized threads, using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters.

[0090] The operation of processing cluster 1214 can be controlled via a pipeline manager 1232, which distributes processing tasks to parallel SIMT processors. Pipeline manager 1232 can receive instructions from scheduler 1210. Fig. Processing Cluster 1214 receives and manages the execution of these instructions via a graphics multiprocessor 1234 and / or a texture unit 1236. Graphics multiprocessor 1234 can be an example instance of a parallel SIMT processor. However, different types of parallel SIMT processors of different architectures can be contained within Processing Cluster 1214. One or more instances of graphics multiprocessor 1234 can be contained within Processing Cluster 1214. Graphics multiprocessor 1234 can process data, and a data crossbar 1240 can be used to distribute processed data to one of several possible destinations, including other shader units. Pipeline Manager 1232 can facilitate the distribution of processed data by specifying destinations for processed data to be distributed via Data Crossbar 1240.

[0091] Each graphics multiprocessor 1234 within processing cluster 1214 can include an identical set of functional execution logic (e.g., arithmetic logic units, load / store units, etc.) to perform computations for any of the operations described above or elsewhere herein. Functional execution logic can be configured in a pipelined manner, allowing new instructions to be issued before previous instructions have completed. Functional execution logic can support a wide variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and the computation of various algebraic functions. The same hardware functional unit can be used to perform different operations, and any combination of functional units can be present.

[0092] Instructions sent to processing cluster 1214 can constitute a thread, which may also be called a warp, subgroup, wave, or wavefront. A set of threads running across a set of parallel processing engines can be called a thread group. A thread group can execute a common program on different input data. Each thread within a thread group can be assigned to a different processing engine within graphics multiprocessor 1234. A thread group can contain fewer threads than the number of processing engines within graphics multiprocessor 1234. If a thread group contains fewer threads than the number of processing engines, one or more of the processing engines may be idle during cycles in which that thread group is being processed.A thread group can contain more threads than the number of processing engines within a 1234 graphics multiprocessor. If a thread group contains more threads than the number of processing engines within a 1234 graphics multiprocessor, processing can be performed over successive clock cycles. Multiple thread groups can run concurrently on a 1234 graphics multiprocessor.

[0093] The 1234 graphics multiprocessor includes an internal cache memory for performing load and store operations, such as, but not limited to, any of the operations described above or elsewhere herein. The 1234 graphics multiprocessor may forgo an internal cache and instead use a cache memory (e.g., the 1248 L1 cache) within processing clusters 1214. Each 1234 graphics multiprocessor may also access L2 caches within partitioning units (e.g., partitioning units 1220A-1220N of Fig. 12A) which can be shared by all processing clusters 1214 and used to transfer data between threads. Graphics multiprocessor 1234 can also access off-chip global memory, which may include one or more local parallel processor memory and / or system memory. Any memory outside of parallel processing unit 1202 can be used as global memory. Processing cluster 1214 can contain multiple instances of graphics multiprocessor 1234 and can share common instructions and data, which can be stored in L1 cache 1248.

[0094] Each processing cluster 1214 can contain an MMU 1245 (memory management unit) that can be configured to map virtual addresses to physical addresses. One or more instances of MMU 1245 can be located within memory interface 1218. Fig. MMU 1245 can reside in 12A. MMU 1245 can include a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile and, optionally, a cache row index. MMU 1245 can include address translation lookaside buffers (TLBs) or caches that may reside within graphics multiprocessor 1234, L1 cache 1248, or processing cluster 1214. A physical address can be processed to distribute surface data access locally, allowing efficient request nesting between partitioning units. A cache row index can be used to determine whether a request for a cache row is a hit or a miss.

[0095] A processing cluster 1214 can be configured such that each graphics multiprocessor 1234 is coupled to a texture unit 1236 for performing texture mapping operations, such as determining texture sampling positions, reading texture data, and filtering texture data. Texture data can be read from an internal texture L1 cache (not shown) or from an L1 cache within the graphics multiprocessor 1234 and retrieved from an L2 cache, local parallel processor memory, or system memory as needed. Each graphics multiprocessor 1234 can output processed tasks to a data crossbar 1240 to provide a processed task to another processing cluster 1214 for further processing or to store a processed task in an L2 cache, local parallel processor memory, or system memory via a memory crossbar 1216.A preROP 1242 (preraster operations unit) can be configured to receive data from graphics multiprocessor 1234 and to route data to ROP units that may be located with partitioning units as described herein (e.g., partitioning units 1220A-1220N of . Fig. 12A). PreROP 1242 can perform optimizations for color mixing, organizing pixel color data, and performing address translations.

[0096] In at least one embodiment, processing cluster 1214 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein.

[0097] Fig. Figure 12C shows a graphics multiprocessor 1234 according to at least one embodiment for performing, for example, any of the operations described above or elsewhere herein. The graphics multiprocessor 1234 can be coupled to a pipeline manager 1232 of a processing cluster 1214. The graphics multiprocessor 1234 can include an execution pipeline, including, but not limited to, an instruction cache 1252 (which can, for example, store instructions such as, but not limited to, compiled API instructions), an instruction unit 1254, an address mapping unit 1256, a register file 1258, one or more general-purpose graphics processing unit (GPGPU) cores 1262, and one or more load / store units 1266, wherein one or more load / store units 1266 can perform load / store operations to load / store instructions according to the execution of an operation.GPGPU cores 1262 and load / store units 1266 can be coupled via a memory and cache link 1268 with cache memory 1272 and shared memory 1270. GPGPU cores 1262 can be part of a SoC, such as, but not limited to, part of the integrated circuit 1100 in [reference missing]. Fig. 11.

[0098] Instruction Cache 1252 can receive a stream of instructions (e.g., to perform any of the operations described above or elsewhere herein) for executing Pipeline Manager 1232. Instructions can be cached in Instruction Cache 1252 and dispatched for execution by an Instruction Unit 1254. Instruction Unit 1254 can dispatch instructions as thread groups (e.g., warps, subgroups, wavefronts, or waves), with each thread in a thread group being assigned to a different execution unit within GPGPU Cores 1262. An instruction can access any address space from a local, shared, or global address space by specifying an address within a unified address space.Address mapping unit 1256 can be used to translate addresses in a unified address space into a different memory address that can be accessed by load / store units 1266.

[0099] Register file 1258 can provide a set of registers for functional units of the graphics multiprocessor 1234. Register file 1258 can provide temporary storage for operands associated with data paths of functional units (e.g., GPGPU cores 1262, load / store units 1266) of the graphics multiprocessor 1234. Register file 1258 can be subdivided among each of the functional units, so that each functional unit is allocated a dedicated portion of register file 1258. Register file 1258 can be subdivided among different warps (which can be referred to as wavefronts, subgroups, and / or waves or threads) executed by the graphics multiprocessor 1234.

[0100] GPGPU 1262 cores can each contain floating-point units (FPUs) and / or arithmetic logic units (ALUs) that can be used to execute instructions from the 1234 graphics multiprocessor. GPGPU 1262 cores can be similar or different in architecture. One set of GPGPU 1262 cores can contain a single-precision FPU and an integer ALU, while a second set of GPGPU cores can contain a double-precision FPU. FPUs can implement the IEEE 754-2008 standard for floating-point arithmetic or enable variable-precision floating-point arithmetic. The 1234 graphics multiprocessor can additionally include one or more fixed-function or special-purpose function units to perform specific functions, such as, but not limited to, copy-rectangle or pixel-mixing operations. One or more of the GPGPU 1262 cores can also contain fixed or special function logic.

[0101] GPGPU Cores 1262 can include SIMD logic capable of executing a single instruction on multiple sets of data. GPGPU Cores 1262 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions for GPGPU Cores can be generated at compile time by a shader compiler or automatically when running programs written and compiled for Single-Program-Multiple-Data (SPMD) or SIMT architectures. Multiple threads of a program can be configured for a SIMT execution model that can be executed via a single SIMD instruction. For example, eight SIMT threads performing the same or similar operations can run in parallel over a single SIMD8 logic unit.

[0102] Memory and cache association 1268 can include a network that connects each functional unit of the graphics multiprocessor 1234 to the register file 1258 and to shared memory 1270. Memory and cache association 1268 can be a crossbar association, allowing the load / store unit 1266 to perform load and store operations between shared memory 1270 and the register file 1258. The register file 1258 can operate at the same frequency as GPGPU cores 1262, thus enabling very low-latency data transfer between GPGPU cores 1262 and the register file 1258. Shared memory 1270 can be used to enable communication between threads running on functional units within the graphics multiprocessor 1234. Cache memory 1272 can, for example, be used as a data cache to temporarily store texture data that is communicated between functional units and texture unit 1236.Shared memory 1270 can also be used as a program-managed cache. Threads running on GPGPU cores 1262 can programmatically store data within shared memory in addition to automatically cached data stored within cache memory 1272.

[0103] A parallel processor or GPGPU, as described herein, can be communicatively coupled to host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. A GPU can be communicatively coupled to host processor / cores via a bus or other interconnect (e.g., a high-speed interconnect such as PCIe or NVLink, but not limited to). A system-on-a-chip (SoC) can incorporate a parallel processor or GPGPU, as described herein, with the parallel processor or GPGPU running on the SoC. A GPU can be integrated as cores on a package or chip and communicatively coupled to cores internally to a package or chip via an internal processor bus / interconnect.Regardless of how a GPU is connected, processor cores can assign work to such a GPU in the form of sequences of instructions contained in a work descriptor. The GPU can then use dedicated switching technology / logic to efficiently process these instructions to perform any of the operations described above or elsewhere herein.

[0104] In at least one embodiment, the graphics multiprocessor 1234 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein.

[0105] Fig. Figure 13 shows a Processor 1300 according to at least one embodiment. Processor 1300 may include a hybrid-architecture processor (e.g., Lunar Lake or Meteor Lake) from Intel Corporation in Santa Clara, CA, or another processor that shares at least some of the components described herein. Processor 1300 may include one or more central processing units (CPU 1302), one or more graphics processing units (GPU 1306), and / or one or more neural processing units (NPU 1304), which may, for example, be a dedicated AI accelerator that offloads artificial intelligence (AI) workloads from the CPU and GPU. Processor 1300 may use instructions that, when executed, cause Processor 1300 and / or any of its components to perform some or all of the processes and techniques described elsewhere herein.The Processor 1300 can include any number of Memory and Cache Units 1310 to facilitate processing among the various components. Memory and Cache 1310 on the Processor 1300 can include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. With respect to the Processor 1300 and any of its components described above or elsewhere herein, one or more of the APIs described herein can, for example, be compiled into instructions that are retrieved by instruction-fetching logic or equivalent, decoded by a processor decoder or equivalent, and scheduled for execution by a scheduler or equivalent (e.g.,APIs (in order or out of order) can be executed by execution logic or equivalents, reordered, and then shut down by shutdown logic or equivalents. APIs (and / or compiled instructions including APIs) can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. One or more of the APIs described herein can include a call.

[0106] The 1300 processor can incorporate compute engines as CPUs 1302 and can include any number of cores, such as, but not limited to, up to 16 cores / 22 threads. Cores in CPU 1302 can be P cores (Performance), E cores (Efficient), and LP-E cores (Low-Power Efficient). Performance cores can be used for single-threaded, computationally intensive workloads with low latency, while efficient cores can be used for multi-threaded, less computationally intensive workloads. Low-Power Efficient cores can be used for scalable multi-threaded performance and offloading background tasks. P cores can be used for single- and limited-threaded performance, whereas E and LP-E cores can be used for multi-threaded throughput and power efficiency.

[0107] The GPU 1306 can include any number of graphics engines, such as, but not limited to, Intel® Arc™ Graphics Engines (Xe LPG) with 8 Xe cores (up to 128 execution units or EUs). As shown in Fig. As shown in Figure 13, the GPU 1306 can include vector engines 1310 and matrix engines 1312, which can, for example, execute FP, INT, and matrix operations all at the same time, separately, or in batches. The GPU 1306 can include a load / store unit 1314 as well as other memory, such as, but not limited to, an instruction cache (I$) 1316 and L1 cache / subsystem local memory (SLM) 1318, which can, for example, store instructions to perform any of the operations described above or elsewhere herein.

[0108] The NPU 1304 can include one or more Intel® AI Boost built-in neural processing units (NPUs). The NPU 1304 can be listed as an integrated PCIe device for the host processor. The NPU 1304 can include one or more (e.g., two) NCE (Neural Compute Engine) tiles 1330. Each tile can be configured with any combination of, but not limited to, (e.g., 2000) Multiply Accumulate (MAC) Engines 1334, a post-processing engine (not shown), an AI DSP processor (not shown), and memory (2 MB dedicated SRAM) per tile, as shown in Fig. Figure 13 shows that for general-purpose computing needs, Neural Compute Engines 1330 can include Streaming Hybrid Architecture Vector Engines (SHAVE) 1328 for high-performance parallel computing, which can include DMA (Direct Memory Access) Engines 1324 to switch data between system memory DRAM (Dynamic Random Access Memory) 1326 and a software-managed cache. An integrated MMU (Memory Management Unit) 1322 plus IOMMU (Input / Output Memory Management Unit) (not shown) can support multiple concurrent hardware contexts and provide security isolation between execution contexts according to the MCDM (Microsoft Compute Driver Model) architecture.The 1300 processor may also include a media unit (not shown) that is included on or separate from the XCDs or other components of the processor to enable video playback and video processing of compressed or uncompressed data, such as using HEVC, AV1, VP9 and AVC HW accelerated decoding support and HEVC, VP9 and AVC HW accelerated encoding support.

[0109] An Intel® Thread Director, which includes firmware built into the processor, can prioritize and manage workload distribution, sending tasks to optimized cores. For example, Thread Director can link P cores, E cores, and / or LP-E cores (described above) along with task scheduling capabilities and the ability to send less demanding tasks to the E cores or LP-E cores. Intel® Deep Learning Boost (Intel® DL Boost) (not shown) can provide built-in AI acceleration for training and inference workloads and can include VNNI (for CPU) and DP4a (for GPU) instruction set support. This instruction set can be optimized with OpenVINO™ Toolkit and oneAPI to accelerate INT8 inference. A software stack, such as that described elsewhere herein, can be used to enable AI inference using OpenVINO™ Toolkit.The 1300 processor can be configured to run an application program, such as, but not limited to, a CUDA program.

[0110] In at least one embodiment, processor 1300 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0111] Processor 1300 may alternatively include a processor based on Qualcomm Corporation's AI Engine Direct architecture in Santa Clara, CA, or another processor that shares at least some of the components described herein, and may include any number of NPUs, GPUs, CPUs, and other related components, such as, but not limited to, NPU 1304 as a Hexagon NPU, GPU 1306 as an Adreno GPU, CPU 1302 as a Kryo or Qualcomm Oryon CPU, as well as a Qualcomm Sensing Hub (not shown) and a Memory Subsystem 1310 in any combination. NPU 1304 may include a power rail, a microtile inference unit, a hardware acceleration unit, a tensor unit, a scalar unit, and a vector unit (all not shown), which may include dedicated or shared memory (e.g., cache or memory such as HBM3) for, e.g.,They may contain instructions for performing any of the operations described above or elsewhere herein. The Adreno GPU 1306 can provide graphics and parallel processing for AI in formats such as, but not limited to, 32-bit floating point (FP32), 16-bit floating point (FP16), and 8-bit integer (INT8). Kryo or Qualcomm Oryon CPUs 1302 can perform AI workloads and can handle contextualization for ubiquitous generative AI applications. The CPU 1302 may also include an instruction fetch unit, a rename and withdraw unit, a memory management unit, a vector execution unit, an integer execution unit, and a load and store unit for processing and instruction management.With respect to Processor 1300 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that may be fetched by the Instruction Fetch Unit, decoded by a Processor Decoder or equivalent, scheduled for execution by a Scheduler or equivalent (e.g., sequentially or out of sequence), executed by Execution Logic or equivalent, reordered, and then shut down by the Rename and Retract Unit. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the Processor (e.g., in cache and / or memory).Any number of CPU 1302 cores can be contained in any number of CPU clusters, which can be coupled with memory and / or cache, such as, but not limited to, a shared L2 cache. Memory can be separate or shared; for example, CPU clusters of CPU 1302 cores can be coupled with a Memory Subsystem 1310, which can include fabric, system-level cache, and any number of memory management units capable of reading and writing memory (e.g., DRAM). The Qualcomm Sensing Hub (not shown) includes micro-NPUs, a power rail, and conventional sensors (a gyroscope, accelerometer, even a barometer) carrying voice and data streams. Memory subsystem 1310 can include memory and cache on processor 1300, which may include one or more levels of cache (e.g., L1, L2, L3 and / or last-level cache) and high-bandwidth memory (e.g.,HBM2e or HBM3) in any combination, for example, to store information and / or instructions for performing any of the operations described above or elsewhere herein. All or part of the memory and / or cache in Memory Subsystem 1310 can be shared or used individually by any one or combinations of components (e.g., GPU 1306, NPU 1304, and CPU 1302) on Processor 1300.

[0112] The Qualcomm AI Engine 1300 can be programmed and controlled with a software stack to perform some or all of the operations described herein. This stack includes, for example, a Qualcomm® Neural Processing SDK for inference, with versions for Android, Linux, and Windows. Developer libraries and services support the latest programming languages, virtual platforms, and compilers. At a lower level of the software stack, system software includes the basic real-time operating system (RTOS), system interfaces, and drivers. The software stack supports various operating systems, including Android, Windows, Linux, and QNX, as well as deployment and monitoring infrastructure such as Prometheus, Kubernetes, and Docker. OpenCL and DirectML are supported for direct cross-platform access to the GPU. For the CPU, LLVM compiler infrastructure optimizations enable accelerated and efficient AI inference.With respect to Qualcomm AI-Engine 1300 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., sequentially or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory). A result of API(s) may then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory.

[0113] In at least one embodiment, Processor 1300 or Qualcomm AI Engine 1300 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to execute one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0114] Fig. Figure 14A illustrates a Processor 1400 according to at least one embodiment. Processor 1400 may include a processor from Intel Corporation's Scalable Family of processors in Santa Clara, CA, or another processor that shares at least some of the components described herein. Processor 1400 may include one or more cores 1412(1)-1412(N), where N is an integer greater than 1 capable of performing the operations described elsewhere herein. Cores 1412(1)-1412(N) may be interconnected using ring and / or mesh interconnects. The mesh interconnect architecture allows an array of vertical and horizontal communication paths to enable traversal from one core to another 1412(1)-1412(N) via a shortest path (jumping on the vertical path to correct row and jumping on the horizontal path to correct column).For mesh interconnects, a chip can accommodate cores 1412(1)-1412(N) and can include a grid of converged mesh stops (CMS) that can be associated with cores 1412(1)-1412(N) (e.g., 1:1). Each core can be associated with a lower-level cache (LLC) slice 1414(1)-1414(N), or cores 1412(1)-1412(N) can share cache, e.g., lower-level cache. LLCs 1414(1)-1414(N) can be inclusive, by including blocks in higher-level cache (e.g., L2 cache), or non-inclusive (with blocks that may not be present in higher-level cache). Each core and LLC slice can include a caching and home agent (CHA) (not shown) that can maintain cache coherence by providing resource scalability over mesh links for Intel® Ultra Path Interconnect (Intel® UPI 1416) cache coherence functionality.UPI 1416 can provide a coherent connection for scalable systems and can allow multiple processors to share a single address space through connections such as, but not limited to, two or three UPI connections per processor.

[0115] The 1400 processor may also include the 1410 system agent, which can accommodate and / or perform various functionalities, such as, but not limited to, memory management, display functions, and / or input / output (I / O) functions. For example, the 1400 processor may include one or more 1408 integrated memory controllers (IMCs). The 1408 IMC can control and manage memory, such as, but not limited to, various memory types, e.g., DDR RAM, such as DDR4, or others described elsewhere herein. The 1410 system agent may include a display controller (not shown) to support display(s). The 1410 system agent may also include PCIe 1404 (e.g., up to 20 PCIe lanes), which can connect, for example, to an external dedicated graphics hookup via DMI bus (e.g., Intel's DMI 3.0 bus) 1406.Systemagent 1410 can include an image processing unit (IPU) (not shown) that incorporates an image signal processor (ISP) on the chip. Fabric 702 can provide scalability for the connection.

[0116] Fig. Figure 14B illustrates components within the core 1412 according to at least one embodiment. The core 1412 may include a frontend 1418, a backend or execution engine 1432, and a memory subsystem 1442. The frontend 1418 may provide the execution engine 1432 with operations (e.g., operations described elsewhere herein) by decoding instructions stored in memory. For example, the frontend 1418 may include a micro-operation (µOps) cache path and / or a legacy path, along with a branch prediction unit 1420 that can determine path instructions. A legacy path for instructions may involve retrieving variable-length instructions (e.g., x86) from the L1 instruction cache, queuing the instructions into instruction queue 1424, and decoding instructions using decoder 1426 into µOps, which can then be provided to assignment queue 1428.Alternatively, a µOPs cache path can include a cache containing pre-decoded µOps (µOps 1430) that can be sent to the assignment queue 1428. Assignment queue 1428 can act as an interface between frontend 1418 and execution machine 1432 and can provide instructions to execution machine 1432. For example, one or more of the API(s) described herein can be compiled into instructions that can be stored, processed, and executed by frontend 1418 and execution machine 1432 and stored in storage subsystem 1442.

[0117] Execution machine 1432 can receive micro-operations in reorder buffer 1434, which can register assignments, rename, and withdraw micro-operations. From the reorder buffer, micro-operations can be sent to scheduler 1436, which can be connected to one or more different execution units 1438. Execution units 1438 can perform, for example, basic arithmetic logic unit (ALU) operations, multiplication, division, and / or more complex operations, such as, but not limited to, various vector operations. Scheduler 1436 can manage the queuing of micro-operations for one or more execution units 1438, for example, depending on the operations that need to be performed.

[0118] Storage subsystem 1442 can handle load and store requests as well as ordering operations. For example, micro-operations (µOPs) can relate to memory access (e.g., load and store), and these can be sent to dedicated scheduler ports that can perform these memory operations. Store and load operations can, for example, be sent to load and store buffer(s) 1444. Storage subsystem 1442 can also include shared or separate L1 data and instruction cache 1446 and L2 cache 1448, which can be used and shared by L1 data and instruction cache 1446. As above for Fig. As described in 14A, each 1412 core can be connected to a slice of a third-level cache (e.g., LLC 1414) that can be shared by all 1412 cores.

[0119] In at least one embodiment, processor 1400 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0120] Fig. Figure 15 illustrates an AI Accelerator 1500 according to at least one embodiment. Processor 1300 may include a processor with AI Accelerator architecture from Intel Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. AI Accelerator 1500 may use instructions which, when executed by AI Accelerator 1500, cause AI Accelerator 1500 to perform some or all of the processes and techniques described elsewhere herein. With respect to AI Accelerator 1500 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that are retrieved by instruction-fetching logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g.,APIs (and / or compiled instructions including APIs) can be executed (in order or out of order) by execution logic or equivalents, reordered, and then shut down by shutdown logic or equivalents. APIs can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. The AI ​​Accelerator 1500 can include one or more compute chips, which can contain homogeneous or heterogeneous processors. Compute chips can contain one or more central processing units (CPUs), one or more graphics processing units (GPUs), or combinations of both.

[0121] In at least one embodiment, computing chips can include computing machines for performing AI calculations. In at least one embodiment, computing chips of AI Accelerator 1500 can be divided into any number (e.g., four) clusters, which can be designated as DCORE (Deep Learning Core) 1506 and contain any number of Matrix Multiplication Machines (MMEs) 1508, Tensor Processing Cores (TPCs) 1510, and L2 Cache 1514 in any combination. MME 1508 can perform operations that use matrix multiplication, such as fully connected layers, convolutions, and batch general matrix multiplications (GEMMs).MME 1508s can be equipped with multiply-accumulate units (MACs) (not shown) that can perform operations such as general matrix multiplication (GEMM), including, but not limited to, an AxB multiplication, which involves generating tensor C[NxM] from two input tensors, A[NxK] and B[KxN]. MME 1508s can be programmed with array dimensions, locations, data types, and various execution operands. MME 1508s can retrieve tensors A and B from memory by pulling them into their streaming buffers so that the matrix multiplication is performed in parallel by the MACs. MME 1508s can push tensor C back into memory after completion.TPC(s) 1510 can include any number of scalar units for performing scalar operations, any number of vector units for performing vector operations, any number of register files or local memory units (e.g., a local vector memory), and instruction load and store components that can be coupled with memory or cache (e.g., HBM, L3 cache, and / or L2 cache) (all not shown). TPCs can support various types of parallel processing, such as Very Long Instruction Word (VLIW) and Single-Instruction-Multiple-Data (SIMD), which supports data types such as, but not limited to, FP32, BF16, FP16 & FP8 (both E4M3 and E5M2), UINT32, INT32, UINT16, INT16, UINT8, and INT8. Any number of compute chips can be interconnected by a cluster. A connection between the computing chips can be established via an interposer bridge, which is transparent to software, for example.

[0122] Memory on the AI ​​Accelerator 1500 can include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. Memory and / or cache systems can be unified or separate. Compute chips of the AI ​​Accelerator 1500 can include on-chip memory containing one or more levels (e.g., two levels) of cache. On-chip SRAM or other memory described elsewhere herein can be used as a unified, last-level (L3) cache or can be divided into slices of L2 cache accessible to groups of MMEs 1508 and TPCs 1510. The use of on-chip memory as L2 or L3 cache can be fully configurable by software, which can dynamically determine its optimal cache allocation per I / O tensor.The AI ​​Accelerator 1500 can include one or more Memory Management Units (MMUs) 1522 for managing memory, such as enabling the AI ​​Accelerator 1500 memory subsystem to operate in a virtual space when accessing VRAM.

[0123] The AI ​​Accelerator 1500 can include a communication port (e.g., a PCIe Gen5 x16 port) 1502 for communicating with a host and a scheduling and synchronization unit 1504. The AI ​​Accelerator 1500 can include a media unit 1516, which can contain any number or combination of media decoder engines (DECs) 1520 and rotator engines (ROTs) 1518. The AI ​​Accelerator 1500 can include a network unit 1524, which can contain any number or combination of network ports 1526 and the accompanying RDMA engine(s) 1528, L2 cache, and memory (e.g., HBM2e or HBM3) stacks. The AI ​​Accelerator 1500 can include a programmable control path entity (not shown) to manage the parallel and efficient execution of different machines.The control path can include submission queues (SQs) that can be issued by the runtime system, completion queues (CQs) that can be used for job completion reporting, a programmable scheduling mechanism that can be used for task scheduling, a programmable hardware synchronization mechanism or "Sync Manager (SM)" that can be used for hardware synchronization, and a programmable interrupt service mechanism or "Interrupt Manager (INTR)" that can enable the passing of asynchronous events to drivers.

[0124] The AI ​​Accelerator 1500 can include media decoding units that support video formats such as HEVC, Progressive H.264, SVC Base Layer, MVC, VP9, ​​JPEG, and Progressive JPEG, among others. The AI ​​Accelerator 1500 can support post-processing of decoded media streams, including but not limited to image downscaling (resizing the image), vertical and horizontal scaling at various ratios, image upscaling, image cropping, bilinear scaling, and Lancos scaling. The AI ​​Accelerator 1500 can implement two post-processing channels per decoder unit: one with scalar (top and bottom) and one for outputting the original image only.The AI ​​Accelerator 1500 can include a hardware rotator machine that performs the following transformations of an input image: 2D rotation, 3D rotation, projection, distortion and undistorting of images, re-sampling of input data at user-defined coordinates, and re-scaling.

[0125] RDMA 1528 over converged Ethernet to AI Accelerators 1500 enables scaling from a single node (i.e., a single AI Accelerator 1500) to hundreds or thousands of nodes or AI Accelerators 1500. The NW subsystem 1524 can include an Intel® Gaudi® Communication Library (IGCL), a main conductor that orchestrates data movement, and a programmable scheduling mechanism that enables smooth machine activation while maintaining task dependencies. An accelerator network subsystem can include Gigabit Ethernet NIC ports 1526, a Layer 2 MAC (not shown), and RDMA engines 1528. AI Accelerators 1500 can include aggregation machines for performing summation activities. All engines in the processor 1500 can operate in parallel; for example, the MME 1508, TPC 1510, and NIC 1526 can all operate concurrently. There can be dependencies between operations that run on different machines, e.g.For example, the output of one machine can be used as the input of another, and / or it can be scheduled for MME, TPC, and NIC to run in parallel. When one machine has completed its execution operation, another machine can be scheduled to begin work on the next operation (as soon as its inputs are ready).

[0126] The AI ​​Accelerator 1500 can be operated and controlled using software layer 1528, which may include low-level components such as, but not limited to, a graph compiler, an automatic kernel fuser, and a library of pre-compiled kernels, as well as integration with AI ecosystems such as, but not limited to, PyTorch, DeepSpeed, Hugging Face, vLLM, Ray, and more, or as described elsewhere herein in relation to software and programming platforms. Software layer 1528 may include implementations of algorithms such as, but not limited to, Page Attention, Flash Attention, and more. Software layer 1528 may generate optimized binary code that implements the given model topology, such as, but not limited to, performing operator fusion, data layout management, parallelization, pipelining and memory management, and graph-level optimizations.

[0127] In at least one embodiment, AI Accelerator 1500 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0128] A neuromorphic computing system is described that uses a multi-core architecture, with each core housing the computing elements including neurons, synapses with on-chip learning capability, and local memory for storing synaptic weights and routing tables. Fig. Figure 16 is a simplified block diagram 1600 illustrating an example of at least one part of such a neuromorphic computing device 1605, according to at least one embodiment. The neuromorphic computing device 1605 may include a neuromorphic processor from Intel Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. As shown in this example, a device 1605 may be provided with a network 1610 consisting of multiple neural network cores interconnected by an internal network, such that several different connections may potentially be defined between the cores. For example, a network 1610 of spiking neural network cores may be provided in the device 1605, and each core may communicate via short packetized spike messages sent from core to core over the network channels. Each core (e.g.,1615) may possess processing and memory resources and logic to implement a certain number of primitive nonlinear temporal computational elements, such as, but not limited to, multiple (e.g., 1000+) distinct artificial neurons (referred to herein as "neurons"). For example, each core may be capable of implementing multiple neurons simultaneously, so that the collection of neuromorphic cores may implement many multiples of neurons using the device. With respect to neuromorphic computational device 1605 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that are called by instruction-fetching logic or equivalents, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g.,APIs (and / or compiled instructions including APIs) can be executed (in order or out of order) by execution logic or equivalents, reordered, and then shut down by standby logic or equivalents. APIs can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents.

[0129] Continuing with the example of Fig. 16. A neuromorphic computing device 1605 can additionally include a processor 1620 and system memory 1625 to implement one or more components for managing and providing functionality of the device. For example, a system manager 1630 can be provided to manage global attributes and operations of the device (e.g., attributes affecting the network of cores 1610, multiple cores in the network, connections of the device 1605 with other devices, managing access to global system memory 1625, among other possible examples). In one example, a system manager 1630 can define and provide a specific routing table to the various routers in the network 1610, orchestrate a network definition and attributes (e.g., weights, decay rates, etc.).), which are to be applied in the network, core synchronization and time-division multiplex management, routing of inputs to the appropriate cores, among other possible functions.

[0130] As another example, a neuromorphic computing device 1605 can additionally include a programming interface 1635 through which a user or system can specify a neural network definition to be applied (e.g., by means of a routing table and individual neuron properties) and implemented by the mesh 1610 of neuromorphic cores. A software-based programming tool can be provided with or separate from the neuromorphic computing device 1605, through which a user can provide a definition for a specific neural network to be implemented using the network 1610 of neuromorphic cores. The programming interface 1635 can take the programmer's input and then generate appropriate routing tables and local memory for individual neuromorphic cores (e.g.,1615) with the specified parameters to implement a corresponding, customer-specific network of artificial neurons implemented by the neuromorphic nuclei.

[0131] In some cases, a neuromorphic computing device 1605 can advantageously be connected to and work compatiblely with other devices, including general-purpose computing devices, to realize specific applications and use cases. Accordingly, in some cases, external interface logic 1640 can be provided to communicate with one or more other devices (e.g., via one or more defined communication protocols). An external interface 1640 can be used to accept input data from another device or an external storage controller acting as the input data source. An external interface 1640 can additionally or alternatively be used to allow results or outputs of computations from a neural network implemented using the neuromorphic computing device 1605 to be transmitted to another device (e.g., a computer, a computer, or a computer).(another general-purpose processor implementing a machine learning algorithm) to enable additional applications and improvements, among other examples.

[0132] As in Fig. Figure 16 shows a network 1610 consisting of multiple neural network cores interconnected by an internal network, illustrating part of a network fabric that connects multiple neuromorphic cores (e.g., 1615a-d). For example, a number of neuromorphic cores (e.g., 1615a-d) can be deployed in a mesh, with each core connected by a network that includes a number of routers (e.g., 1650). In one implementation, each neuromorphic core (e.g., 1615a-d) can be connected to a single router (e.g., 1650), and each router can be connected to at least one other router (as in Figure 1610). Fig. (16 shown). As an example, in a given implementation, four neuromorphic cores (e.g., 1615a-d) can be connected to a single router (e.g., 1650), and each of the routers can be connected to two or more other routers to form a manycore mesh that allows each of the neuromorphic cores to connect to every other neuromorphic core in the device. Furthermore, since each neuromorphic core can be configured to implement multiple different neurons, the device's router network can equally allow connections, or artificial synapses (or simply "synapses"), to be defined between any two of the potentially many (e.g., 30,000+) neurons defined using the network of neuromorphic cores provided in a neuromorphic computing device.

[0133] Fig. Figure 16 shows a block diagram illustrating internal components of an example implementation of a neuromorphic kernel 1615. In one example, a single neuromorphic kernel can implement a certain number of neurons (e.g., 1024) that share the neuromorphic kernel's architectural resources in a time-multiplexed manner. In another example, each neuromorphic kernel 1615 can contain a processor block 1655 capable of performing arithmetic functions and routing in connection with the realization of a digitally implemented artificial neuron, as explained herein, but not limited to.Each neuromorphic core 1615 can additionally provide local memory in which a routing table can be stored and accessed for a neural network. The accumulated potential of each soma of each neuron implemented using the core can be tracked, and the parameters of each neuron implemented by the core can be recorded, among other data and usage. Components or architectural resources of a neuromorphic core 1615 can further include an input interface 1665 for accepting input spike messages generated by other neurons on other neuromorphic cores, and an output interface 1670 for sending spike messages to other neuromorphic cores via the mesh network. In some cases, routing logic for the neuromorphic core 1615 can be implemented, at least partially, using the output interface 1670.Furthermore, in some cases, a single kernel (e.g., 1615) can implement multiple neurons within an exemplary SNN, and some of these neurons can be interconnected. In such cases, spike messages sent between the neurons located on that particular kernel can bypass communication via the neuromorphic computing device's routing fabric and instead be managed locally on that particular neuromorphic kernel.

[0134] Each neuromorphic core can additionally include logic to implement an artificial dendrite 1680 and an artificial soma 1685 (referred to herein simply as "dendrite" and "soma," respectively) for each neuron 1675. The dendrite 1680 can be a hardware-implemented process that receives spikes from the network. The soma 1685 can be a hardware-implemented process that receives the accumulated neurotransmitter amounts of each dendrite for the current time and develops each dendrite and the potential state of the soma to generate outgoing spike messages at appropriate times. A dendrite 1680 can be defined for each connection that receives input from another source (e.g., another neuron). In one implementation, the dendrite 1680 process can receive and handle spike messages as they arrive serially from the network in a time-division multiplexed manner.When spikes are received, the activation of the neuron (tracked using soma 1685 and local memory 1660) can increase. If the neuron's activation exceeds a threshold set for neuron 1675, the neuron can generate a spike message, which is propagated via output interface 1670 to a fixed set of fanout neurons. The network distributes the spike messages to all target neurons, and in response, these neurons can update their activations in a transient, time-dependent manner, potentially causing the activation of some of these target neurons to also exceed corresponding thresholds and trigger further spike messages, as in real biological neural networks.

[0135] As noted above, a neuromorphic computing device can reliably implement a spike-based model of neural computation. Such models can also be referred to as spiking neural networks (SNNs). In addition to neuronal and synaptic state, SNNs also incorporate the concept of time. For example, in an SNN, communication occurs via event-driven action potentials, or spikes, which convey no explicit information other than the spike time, as well as an implicit source-target neuron pair corresponding to the spike transmission. Computation takes place in each neuron as a result of the dynamic, nonlinear integration of the weighted spike input. In some implementations, iteration and dynamic feedback can be integrated within an SNN computational model.Furthermore, a variety of network connectivity models can be assumed to model different real-world networks or relationships, including fully connected (all-to-all) networks, forward-feedback trees, completely random projections, and "small-world" networks, among other examples. A homogeneous, two-dimensional network of neuromorphic kernels, such as, but not limited to, the example of [missing example]. Fig. As shown in Figure 16, the device can advantageously support all these network models. Since all cores of the device can be connected, all neurons defined in the cores can therefore also be fully connected through a certain number of router hops. Furthermore, the device can include fully configurable routing tables to define a variety of different neural networks by allowing the neurons of each core to distribute their spikes to any number of cores in the mesh, thus realizing completely arbitrary connectivity graphs.

[0136] In an improved implementation of a system capable of supporting SNNs, such as, but not limited to, the very large integration (VLSI) hardware device used in the example of Fig. As illustrated in Figure 9, high-speed and reliable circuits can be provided to implement SNNs to model the information processing algorithms used by the brain, but in a more programmable way. For example, while a biological brain can only implement a specific set of defined behaviors, as determined by years of development, a neuromorphic processor device can provide the ability to rapidly reprogram all neural parameters. Accordingly, a single neuromorphic processor can be used to realize a wider range of behaviors than those provided by a single slice of biological brain tissue.This distinction can be achieved by using a neuromorphic processor with neuromorphic design implementations that differ significantly from those of naturally occurring neural circuits.

[0137] As an example, a neuromorphic processor can use time-division multiplexing (TDM) in both the spike communication network and the device's neural machinery to implement SNNs. Accordingly, the same physical switching technology of the processor device can be shared among many neurons to achieve a higher neural density. With TDM, the network can connect N cores with O(N) total wiring length, whereas discrete point-to-point wiring would scale as O(N²), achieving a significant reduction in wiring resources to accommodate planar and non-plastic VLSI wiring technologies, among other examples. In the neuromorphic cores, TDM can be implemented through dense memory allocation, for example, using static random-access memory (SRAM), with shared buses, address decoding logic, and other multiplexed logic elements.The state of each neuron can be stored in the processor's memory, with data describing each neuron's state including the state of each neuron's collective synapses, all currents and voltages across its membrane, among other example information (such as, but not limited to, configuration and other information).

[0138] A neuromorphic processor can adopt a "digital" implementation, which differs from other processors that adopt more "analog" or "isomorphic" neuromorphic approaches. For example, a digital implementation might integrate synaptic current using digital adder and multiplier circuits, in contrast to the analog isomorphic neuromorphic approaches, which accumulate charge on capacitors in an electrically analogous way to how neurons accumulate synaptic charge on their lipid membranes. The accumulated synaptic charge could, for example, be stored for each neuron in local memory of the corresponding nucleus.Furthermore, at the architectural level of an exemplary digital neuromorphic processor, reliable and deterministic operation can be achieved by synchronizing time across the network of cores, ensuring that any two implementations of the design, under the same initial conditions and configuration, produce identical results. Asynchronicity can be maintained at the circuit level to allow individual cores to operate as quickly and freely as possible, while determinism is preserved at the system level. Accordingly, the concept of time as a temporal variable in neural computations can be abstracted, thus separating it from the "wall clock" time used by the hardware to perform the computation.Accordingly, some implementations can provide a time synchronization mechanism that globally synchronizes the neuromorphic nuclei at discrete time intervals. This synchronization mechanism allows the system to complete a neural computation as quickly as the circuitry permits, with a divergence between runtime and the biological time that the neuromorphic system models.

[0139] In operation, the neuromorphic mesh device can start in an idle state, with all neuromorphic cores inactive. Since each core iterates asynchronously through its neurons, it generates spike messages that the mesh link directs to the appropriate target cores containing all the target neurons. Because the implementation of multiple neurons on a single neuromorphic core can be time-division multiplexed, a time step can be defined in which all spikes involving the multiple neurons can be processed and accounted for using the shared resources of a corresponding core. Since each core stops servicing its neurons for a given time step, cores in some implementations can communicate with neighboring cores (e.g.,(Using a handshake) using synchronization messages to flush the mesh of all spike messages in flight, allowing the cores to reliably determine that all spikes have been served for the time step. At this point, all cores can be considered synchronized, allowing them to advance their time step, return to the initial state, and begin the next time step.

[0140] Given this context, and as introduced above, a device (e.g., 1605) can be provided that implements a mesh 1610 of interconnected neuromorphic cores, with the core potentially implementing multiple artificial neurons capable of being interconnected to implement a SNN. Each neuromorphic core (e.g., 1615) can provide two loosely coupled asynchronous processes: an input dendrite process (e.g., 1680) that receives spikes from the network and applies them to the appropriate target dendrite compartments at appropriate future times, and an output soma process (e.g., 1685) that receives the accumulated neurotransmitter amounts of each dendrite compartment for the current time and develops each dendrite and the potential state of the soma membrane, generating outgoing spike messages at appropriate times (e.g., when a soma threshold potential has been reached).It should be noted that from a biological perspective, the dendrite and soma names used here only approximate the role of these functions and should not be interpreted too literally.

[0141] In at least one embodiment, neuromorphic computing device 1605 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0142] Fig. Figure 17 is a block diagram of an embodiment of a multi-node network in which remote memory computation can be implemented, according to any embodiment. System 1700 can represent a network of nodes described herein, which can be used, for example, to perform some or all of the operations described herein. System 1700 can represent a data center. System 1700 can represent a server farm. System 1700 can represent a data cloud or a processing cloud. System 1700 can represent a supercomputer. System 17 can include tens, hundreds, or thousands of nodes. The nodes of System 1700 can include processors, such as, but not limited to, central processing units (CPUs), graphics processing units (GPUs), or any combination of processors described herein, such as, but not limited to, other processors in Fig. 11- Fig. 23. With respect to any of the processors in System 1700 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents. System 1700 can include over nine thousand nodes, with each node containing two Intel Xeon Max processors, six Intel Max series GPUs, and a unified memory architecture such as, but not limited to, that used in Intel Corporation's Intel Aurora supercomputer in Santa Clara, CA, or any other supercomputer sharing at least some of the components described herein.

[0143] One or more Clients 1702 make requests to System 1700 via Network 1704. Network 1704 represents one or more local area networks, wide area networks, or a combination thereof. Clients 1702 can be human or machine clients that generate requests for System 1700 to perform operations. System 1700 executes applications or data processing tasks requested by Clients 1702.

[0144] System 1700 can include one or more racks, which represent structural and interconnect resources to house and connect multiple compute nodes. Rack 1710 can contain multiple nodes 1730. Rack 1710 can host multiple blade components 1720. Hosting can refer to providing power, structural or mechanical support, and interconnection. Blades 1720 can refer to compute resources on printed circuit boards (PCBs), with a PCB housing the hardware components for one or more nodes 1730. Blades 1720 may or may not include a chassis, enclosure, or other "box" besides that provided by Rack 1710. Blades 1720 can include enclosures with exposed connectors for connecting to Rack 1710.System 1700 may or may not include Rack 1710, and each Blade 1720 may include a chassis or enclosure that can be stacked or otherwise reside in close proximity to other Blades and allow connection to Node 1730. System 1700 may contain 10,624 compute Blades, which include 63,744 Intel Max Series GPUs and 21,248 Intel Xeon Max CPUs across 166 Racks.

[0145] System 1700 can include Fabric 1770, which represents one or more intermediate connectors for nodes 1730. Fabric 1770 can include multiple switches 1772, routers, or other hardware to route signals between nodes 1730. Additionally, Fabric 1770 can connect System 1700 to Network 1704 for access by clients 1702. Beyond routing equipment, Fabric 1770 can also be considered to include the cables, connectors, or other hardware to connect nodes 1730 to one another. Fabric 1770 can have one or more associated protocols to manage the routing of signals through System 1700. The protocol(s) depend, at least in part, on the hardware used in System 1700.

[0146] As illustrated, Rack 1710 N can contain Blades 1720. In addition to Rack 1710, System 1700 can contain Rack 1750. As illustrated, Rack 1750 M can contain Blades 1760. M is not necessarily the same as N; thus, it is understood that various different hardware equipment components could be used and coupled together in System 1700 via Fabric 1770. Blades 1760 can be the same as or similar to Blades 1720. Nodes 1730 can be any type of node as described herein and need not all be of the same type. System 1700 is neither restricted to being homogeneous nor restricted to being heterogeneous.

[0147] A node in Blade 1720(0) is illustrated in detail. However, other nodes in System 1700 may be the same or similar. At least some nodes 1730 may be compute nodes with a processor 1732 and memory 1740. A compute node refers to a node with processing resources (e.g., one or more processors) that runs an operating system and can receive and process one or more tasks. At least some nodes 1730 may include storage server nodes with a server as processing resources 1732 and memory 1740. A storage server refers to a node with more storage resources than a compute node, and instead of having processors to perform tasks, a storage server includes processing resources to manage access to the storage nodes within the storage server.

[0148] Node 1730 can include an interface controller 1734, which can represent logic to control access by Node 1730 to Fabric 1770. This logic can include hardware resources to connect to the physical linking hardware. The logic can also include software or firmware logic to manage the connection. Interface controller 1734 can be a host fabric interface or can include a fabric interface according to any embodiment described herein.

[0149] Node 1730 can contain Memory Subsystem 1740. Memory 1740 can contain Memory Compute Resources (comp) 1742, which represent one or more capabilities through Memory 1740 for performing memory computations. System 1700 enables remote memory operations, such as, but not limited to, the operations described elsewhere herein. Thus, Node 1730 can request memory computations from remote nodes, with data for the computation remaining local to the executing node, rather than being sent via Fabric 1770 or from memory to the Fabric interface. In response to the execution of the memory computation, the executing node can provide a result to the requesting node.

[0150] Processor 1732 can include one or more separate processors. Each separate processor can include a single processing unit, a multi-core processing unit, or a combination thereof. The processing unit can be a primary processor, such as, but not limited to, a CPU (central processing unit), a peripheral processor, such as, but not limited to, a GPU (graphics processing unit), or a combination thereof. Memory 1740 can be or include memory devices and a memory controller.

[0151] The term "storage device" can refer to various types of storage. Storage devices generally refer to volatile memory technologies. Volatile memory is memory whose state (and therefore the data stored on it) is indeterminate when power to the device is interrupted. Non-volatile memory refers to memory whose state is definite, even when power to the device is interrupted. Dynamic volatile memory requires updating the data stored in the device to maintain its state. An example of dynamic volatile memory includes DRAM (Dynamic Random Access Memory) or a variant such as synchronous DRAM (SDRAM).A memory subsystem as described herein may be compatible with a number of memory technologies, such as, but not limited to, DDR3 (Double Data Rate Version 3, originally published by JEDEC (Joint Electronic Device Engineering Council) on 27.June 2007, currently on Publication 21), DDR4 (DDR version 4, original specification, published by JEDEC in September 2012), DDR4E (DDR version 4, extended, currently under discussion by JEDEC), LPDDR3 (Low-Performance DDR version 3, JESD209-3B, Aug 2013 by JEDEC), LPDDR4 (Low-Performance Dual Data Rate (LPDDR) version 4, JESD209-4, originally published by JEDEC in August 2014), WIO2 (Wide I / O 2 (WideI02), JESD229-2, originally published by JEDEC in August 2014), HBM (High-Bandwidth Memory DRAM, JESD235, originally published by JEDEC in October 2013), DDR5 (DDR version 5, currently under discussion by JEDEC), LPDDR5 (currently under discussion by JEDEC), HBM2 (HBM version 2), currently under discussion by JEDEC) or other or combinations of memory technologies and technologies based on derivatives or extensions of such specifications.

[0152] In addition to or as an alternative to volatile memory, in one embodiment, reference to storage devices may refer to a non-volatile storage device whose state is determined even when power to the device is interrupted. In one embodiment, the non-volatile storage device is a block-addressable storage device, such as, among others, NAND or NOR technologies. Thus, a storage device may also include future-generation non-volatile devices, such as, among others, a three-dimensional intersection point (3DXP) storage device, other byte-addressable non-volatile storage devices, or storage devices that use chalcogenide phase-change material (e.g., chalcogenide glass).In one embodiment, the storage device may be or include multi-threshold-level NAND flash memory, NOR flash memory, single- or multi-level phase-change memory (PCM) or phase-change memory with a switch (PCMS), resistive memory, nanowire memory, ferroelectric transistor random-access memory (FeTRAM), magnetoresistive random-access memory (MRAM) incorporating memristor technology, or spin-transfer-torque (STT) MRAM, or a combination of any of the above or other memory types.

[0153] In at least one embodiment, System 1700 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0154] Fig. Figure 18 illustrates an accelerated processing unit 1800 according to at least one embodiment. The accelerated processing unit 1800 may include a processor based on the CDNA architecture of AMD Corporation in Santa Clara, CA, or another processor that shares at least some of the components described herein. The accelerated processing unit 1800 may include one or more accelerator complex chips (XCDs) 1804 for performing operations described elsewhere herein, such as, but not limited to, graphics processing and / or parallel processing, as well as instruction-level parallel computations, including support for a wide range of precisions (INT8, FP8, BF16, FP16, TF32, FP32, and FP64) and sparse matrix data (i.e., sparsity). In some cases, XCDs may be referred to as graphics compute chips (GCDs).The Accelerated Processing Unit 1800 can include one or more Complex Computing Chips (CCDs) 1806 for performing operations described elsewhere herein, including, but not limited to, those operations performed by host processors. CCDs may in some cases be referred to as core complexes or CCXs, such as CCXs used in AMD Ryzen processors. XCDs and CCDs can share any type of cache or memory (e.g., one or more Memory Units 1802) or have cache or memory allocated to each XCD or CCD or group of XCDs or CCDs. For example, on-package AMD Infinity Fabric combines XCDs and CCDs in shared AMD Infinity Cache 1808 and, in some embodiments, high-bandwidth memory (e.g., HMB3).The Accelerated Processing Unit 1800 can be an AMD MI300a processor, which includes three CPU chiplets (or CCDs) and six accelerator chiplets (XCDs) across four input / output chips (IODs). These can be stacked on a single piece of silicon, which connects them (e.g., via AMD Infinity Fabric) to eight stacks of high-bandwidth DRAM powering the superchip. An AMD MI300x processor replaces the CCDs with two additional XCDs for an accelerator-only system.

[0155] An Accelerated Processing Unit 1800 can include one or more input / output (I / O) interfaces. For example, XCDs 1804 and CCDs 1806 can co-locate on one or more Input / Output Chips (IODs) 1810, which can include one or more I / O interfaces. IODs 1810 can include any number and any type of I / O interface (e.g., PCI, PCI-Extended (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices can be connected to I / O interfaces 1170. I / O interfaces of IODs 1810 can also be used for one or more interconnected Accelerated Processing Units 1800, for example, in a server architecture.

[0156] The Accelerated Processing Unit 1800 can include one or more Memory Units 1802 for storing instructions and other information used to perform operations described elsewhere herein. Memory Units 1802 can include any volatile memory, such as, but not limited to, memory types described elsewhere herein, and can include, for example, high-bandwidth memory (e.g., HMB3) or high-bandwidth DRAM. Memory associated with the Accelerated Processing Unit 1800 (e.g., Memory Units 1802) can include system memory, which can be used, for example, for instructions, statements, constants, and inputs and outputs. Memory Units 1802 can also include device memory, which can be used as storage and, for example, as a return buffer(s) and for private data, for instructions, statements, constants, inputs and outputs.Memory units 1802 can be linked to one or more IODs 1810. In at least one embodiment, L1 cache 1820 initiates a memory hierarchy that includes shared L2 cache 1828, for example, within the XCDs. AMD Infinity Cache™ is a last-level cache (LLC) located on an active I / O chip (IOD). CCDs 1806 and XCDs 1804 can have separate or shared memory. AMD Infinity Architecture and AMD Infinity Fabric™ technology enable coherent unification of GPU and CPU chiplet technologies (e.g., XCDs, CCDs, and / or CCXs) with memory (e.g., stacked HBM3 memory) in single devices and across multiple device platforms with high throughput.

[0157] As in Fig. As shown in Figure 18, an XCD 1804 can include a shared set of global resources 1830, which may include hardware schedulers 1812 and asynchronous computational machines (ACEs) 1824 that send tasks (e.g., compute shader workgroups) to compute units (CUs or cores) 1830. ACEs 1824 (e.g., four) can each be associated with CUs 1830 (e.g., 40 CUs), and some of the CUs may be disabled for yield management. CUs 1830 may have dedicated cache or shared cache (e.g., L2 cache) 1828, which can be used to aggregate all memory traffic for the chip. CUs 1830 can include thread and parallel processor cores, including instruction retrieval and scheduling with Scheduler (S) 1812, Matrix Core Unit (MCU) 1816 and Shader Core (SC) 1818 (e.g., execution units for scalar, vector and matrix data types), as well as load / store pipelines with an L1 cache 1820 and Local Data Section (LDS) 1814.Local data sharing can include, for example, a scratch RAM with built-in arithmetic capabilities that allow data to be shared between threads in a workgroup. An instruction cache 1840 (for example, for storing and providing instructions for performing operations described elsewhere herein) can be associated with one or more CUs and can be shared between two CUs. Matrix cores 1816 can process a variety of data types, such as, but not limited to, INT8, FP8, FP16, BF16, and TF32 data types. An accelerated processing unit 1800 can include compute units 1830, which can be arranged in an array format, for example, as a data parallel processor (DPP) array.The Ultra-Thread Dispatch Processor 1842 can communicate with ACUs 1830, and the Instruction Processor 1844 can read instructions written by the host to memory-mapped registers in a system memory address space (not shown). The Instruction Processor 1844 can send hardware-generated interrupts to a host processor (such as a CCD) when the instruction is complete. The Memory Controller 1836 can also have direct access to all device memory and the host-specified areas of system memory. To meet read and write requests, the Memory Controller 1836 can perform Direct Memory Access (DMA) control functions, including calculating memory address offsets based on the format of the requested data in memory.For example, one or more of the APIs described herein can be compiled into instructions that are stored in Instruction Cache 1840 and then retrieved by Instruction Fetch Logic in Processor 1840, decoded by a Processor Decoder or equivalent, scheduled for execution by a Scheduler or equivalent (e.g., sequentially or out of sequence), executed by Execution Logic or equivalent, reordered, and then quiesced by Quit Logic or equivalent. API(s) (and / or compiled instructions including API(s)) can be stored in any memory outside or inside Processor 1800 (e.g., in cache and / or memory). A result of API(s) can then be stored in memory inside or outside the Processor, including registers, DRAM, Flash, SRAM, cache, or other memory equivalents.

[0158] An application can include a program that runs on a host processor (e.g., a CCD) and programs called kernels that run on one or more XCDs. Programs can be controlled by host instructions that set internal base addresses and other configuration registers, specify a data domain on which the Accelerated Processing Unit 1800 can operate, invalidate and flush caches on the Accelerated Processing Unit 1800, and cause the Accelerated Processing Unit 1800 to begin executing a program. Kernels can be referred to as programs that are executed by the Accelerated Processing Unit 1800. A kernel can be executed independently on each work element or as groups of work elements, which can be called a wavefront, that can execute the kernel on all work elements in the group (e.g., 64) in a single pass.Computing units 1830 can include a scalar arithmetic logic unit (ALU) that can operate on one value per wavefront (common to all worker elements), a vector ALU that can operate on unique values ​​per worker element, a local data sharing unit 1814 that can allow worker elements within a workgroup to communicate and share data, a scalar memory (not shown) that can transfer data between scalar general-purpose registers (SGPRs) and memory through a cache, and vector memory that can transfer data between vector general-purpose registers (VGPRs) and memory, including sampling texture maps. Core control flow can be handled using scalar ALU instructions, which can include if / else branches and loops. Scalar ALU (SALU) and memory instructions can operate on an entire wavefront and on one or more SGPRs.Vector memory and ALU instructions can operate on all work elements in the wavefront simultaneously.

[0159] In at least one embodiment, the accelerated processing unit 1800 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0160] Fig. Figure 19 illustrates a Processor 1900, such as, but not limited to, a processor based on a Zen architecture (such as Zen 1, 2, 3, 4, 5, or others) from AMD Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. Processor 1900 includes one or more CPU chips 1902(1)-1902(N), where N is an integer greater than 1. CPU chip 1902 can include any number of processor cores 1916 (for example, to perform any of the operations described elsewhere herein) and any number of cache memories (for example, to store instructions and other information to perform any of the operations described elsewhere herein) in any combination. For example, L2 cache units 1918 can be coupled with processor core(s) 1916, which can share L2 cache units 1918 and / or be coupled to them individually.Processor cores 1916 can individually couple L3 cache 1922 and / or share L3 cache, which can be a low-level cache (LLC) 1922 for accessing data and other information used by the processor cores 1916. One or more processor cores 1916 and one or more L2 cache units 1918 can be contained in a core complex (CCX) 1920, which can include (e.g., a 32 MB) shared cache (e.g., L3 cache 1922). Core complexes 1920 can be manufactured on a chip (CCD or CPU chip) 1902. For example, up to 12 core complexes 1920, together with 8 CPU chips 1902, can be configured to form a processor, providing up to 96 processor cores 1916 for the processor. For example, a “Zen 4c” core complex 1920 can contain up to eight cores 1916 and a shared 16 MB L3 cache 1922.Two of these 1920 core complexes can be combined on a single 1902 CPU chip for 16 cores per chip and a total of 32 MB of L3 cache per chip. Up to eight of the 1902 CPU chips can be combined with a 1904 I / O unit to provide CPUs with up to 128 1916 processor cores. Up to four of the "Zen 4c" chips described above can be combined to provide CPUs with up to 64 1916 processor cores.

[0161] The 1900 processor can include a variety of input / output configurations, which are described in more detail herein. The 1904 I / O unit can include one or more 1906 memory controllers that manage memory usage (e.g., DDR5 memory) for the 1900 processor. The 1904 I / O unit can include one or more 1912 SATA disk controllers for managing memory and one or more 1914 Compute Express Link (CXL™) 1.1+ memory controllers, which provide CPU-to-device and CPU-to-memory links and can be flexibly assigned specific functions at server design time. The 1904 I / O unit can include 1908 PCIe controllers for connecting peripherals and other components connected to the 1900 processor. The I / O unit 1904 can include USB ports 1910 for connecting to other components separate from the processor 1900. CPU chips 1902 can have any number of connections, e.g.Supports one or two connections to I / O Unit 1904. As shown, I / O Unit 1904 includes the components described herein, and I / O Unit 1904 can be an I / O chip that houses several different components. Memory Controller 1906, PCIe Controller 1908, USB Ports 1910, SATA Controller 1912, and / or CXL Controller 1914 can be integrated anywhere within Processor 1900, either separately or in any groups or combinations thereof.

[0162] The 1900 processor can include Infinity Fabric 1924 assemblies (which may be similar to or based on PCIe architectures) that can provide connections between CPUs (e.g., 1902(1)-1902(N) CPU chips), 1926 graphics processor(s), 1932 inference engine(s), and other components in the multi-chip architecture, such as 1928 secure processor(s) and the 1904 I / O unit. One or more AMD Infinity Fabric™ 1910 assemblies can connect to 1902(1)-1902(N) CPU chips and serve as a connection used between CPUs. One or more Infinity Fabric 1910 assemblies can connect each 1902 CPU chip to the 1910 I / O unit.

[0163] In at least one embodiment, Processor 1900 may include central processing units (CPUs) and other associated hardware and software, as described above and further herein. Processor 1900 may also include Graphics Processor(s) 1926. Graphics Processor 1926 may be used for image generation and processing, as well as other computations and operations further described herein. Graphics Processor 1926 may be based on AMD's RDNA 3 or 3.5 architecture in Santa Clara, CA. Graphics Processor 1926 may include Graphics Computing Chips (GCDs) and Memory Cache Chips (MCDs). GCDs may include any number of compute units (CUs) for graphics or other processing, such as operations performed by arithmetic logic units (ALUs), which are further described herein. Graphics Processor 1926 may include L2 cache that can be used by compute units.MCDs (not shown) can contain any number of memory units and can include cache, such as L3 cache, as well as memory interfaces for coupling to memory, such as Memory 1942(1)-(N), where N is an integer. Components within the Graphics Processor 1926 can be interconnected using various approaches, such as Infinity Fabric 1924 interconnects outside or inside the Graphics Processor 1926.

[0164] Inference Engine 1932 can provide neural processing capabilities for Processor 1900 for computational processes used for neural networks, deep learning, and other operations related to artificial intelligence, which are further described herein. Processor 1900 can include a secure processor 1928 for managing the security of the processor, a display controller 1930 for controlling displays, a system management unit 1934 for managing and operating some or all of the components on Processor 1900, multimedia engines 1936 for audio and video operations, a fusion controller hub 1938 for managing USB, SATA, and PCIe connections to the processor, and a sensor fusion hub 1940 for managing sensors, such as accelerometers. Processor 1900 can also include memory 1942(1)-(N), where N is an integer.Memory can include various memory types, such as LPDDR5 and / or DDR5 or others described elsewhere herein.

[0165] To perform the operations described herein, the 1900 processor can include an execution pipeline containing a front end, which may include a cache (e.g., an L1 cache) that stores instructions (not shown). The flow of instructions can be modified by a branch predictor. Instructions can be decoded by a decoder, sent to a back end for execution, and renamed. Instruction fetch and decode pipes can be sent, for example, to integer or floating-point execution operations, which can be scheduled by a scheduler and transferred to vector and / or general-purpose registers. Floating-point multiplier and / or add operations can be processed, and arithmetic logic units (ALUs) can also be used to perform calculations, such as arithmetic and logical operations.Outputs from the compute units can be coupled with a load / store queue, which may be connected to cache, such as L1 cache and / or L2 cache.

[0166] With respect to Processor 1900 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions (e.g., AVX-512 instructions based on a SIMD model) that may be fetched by instruction call logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0167] In at least one embodiment, processor 1900 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0168] Fig. Figure 20 illustrates an example of a 2000 processing core that can implement Arm architecture (e.g., v9.0-A) or another processor that shares at least some of the components described herein. The Neoverse™ V2 Core 2000 can be implemented within a DynamIQ Shared Unit (DSU) cluster via DSU-110 Aggregation 2054 for one or more interconnected cores, for example, for parallel processing. The Neoverse™ V2 Core can be implemented as a single core in a DSU cluster configured to connect directly, with or without L3 cache, snoop filter, or snoop control unit (SCU) logic (not shown). The Neoverse™ V2 core can include a CPU bridge 2052, which connects core 2000 to a DSU-110 array, which can also connect core 2000 to an external memory system and the rest of a system-on-a-chip. The L1 instruction storage system 2002 can retrieve instructions from an instruction cache 2004 and execute the instructions (e.g.,deliver one or more APIs described herein (which may be compiled into instructions) to an Instruction Decoding Unit 2010, for example, to perform some or all of the operations described above or elsewhere herein. The L1 Instruction Storage System 2002 may include the L1 Instruction Cache 2004, for example, with 64-byte cache lines; the L1 Instruction Translation Lookaside Buffer (TLB) 2006, for example, with native support for 4KB, 16KB, 64KB, and 2MB page sizes; and the Macro Operations Cache (MOP) 2008 (for example, 1536-entry, 4-way skewed associative L0 MOP cache), which may contain decoded and optimized instructions for higher performance. The Instruction Decoding Unit 2010 can decode AArch64 instructions into an internal format. The Register Renaming Unit 2012 can perform register renaming to facilitate out-of-order execution and sends decoded instructions to various output queues.The Instruction Output Unit 2014 can control when decoded instructions can be sent to the execution pipelines, and it can include output queues to store instructions until they are sent to execution pipelines. The Integer Execution Pipeline 2016 can be included in an execution pipeline and can include the Integer Execution Unit 2018, which can perform arithmetic and logical data processing operations. The Vector Execution Unit 2020 can be included in an execution pipeline and can perform Advanced SIMD and Floating Point Operations (FPU) 2022, execute Scalable Vector Extension (SVE) and Scalable Vector Extension 2 (SVE2) instructions 2024, and can optionally execute Cryptographic Instructions (Crypto) 2026. Advanced SIMD can include media and signal processing architecture, adding instructions primarily for audio, video, 3D graphics, image, and speech processing.A floating-point architecture provides support for single-precision and double-precision floating-point operations. The L1 Data Storage System 2030 can execute load and store instructions as well as service memory coherence requests. The L1 Data Storage System 2030 can include an L1 Data Cache 2032 and a fully associative L1 Data TLB 2034 with native support for 4KB, 16KB, and 64KB page sizes and 2MB and 512MB block sizes. The Memory Management Unit (MMU) 2028 can provide fine-grained memory system control through a set of virtual-to-physical address mappings and memory attributes that can be held in translation tables, which can be stored in TLB 2034 when an address is translated. L2 storage system 2036 can include L2 cache 2038 and can be connected to DSU-110 2054 via an asynchronous CPU bridge 2052.The Neoverse™ V2 Core 2000 can support a range of debugging, testing, and tracing options, including a Trace Unit 2042, a Trace Buffer 2040, and an Embedded Logic Analyzer (ELA) 2048. The Neoverse™ V2 Core 2000 can implement the Statistical Profiling Extension (SPE) 2044 to provide a statistical view of the performance characteristics of executed instructions, which software writers can use to optimize their code for better performance. The Power Monitoring Unit (PMU) 2046 can provide performance monitoring that can be configured to collect statistics on the operation of each core and the memory system. This information can be used for debugging and code profiling. The Generic Interrupt Controller (GIC) CPU Interface 2050, when integrated with an external distributor component, can be a resource for supporting and managing interrupts in a cluster system.In a cluster, there can be a CPU Bridge 2052 between each Neoverse™ V2 Core 2000 and DSU-110 2054. CPU Bridge 2052 can control buffering and synchronization between Core 2000 and DSU-110 2054. CPU Bridge 2052 can be asynchronous to allow different frequency, power, and area implementation points for each Core 2000. CPU Bridge 2052 can run synchronously without affecting other interfaces, such as, but not limited to, debug and trace, which can be asynchronous.

[0169] In at least one embodiment, Kern 2000 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0170] Fig. Figure 21 illustrates one or more chips incorporating one or more tensor processing units (TPUs) 2100 according to at least one embodiment. TPUs 2100 in Fig. TPUs 2100 can include application-specific integrated circuits (ASICs), for example, to perform some or all of the operations described above or elsewhere herein, such as, but not limited to, accelerating machine learning workloads that perform matrix operations. TPUs 2100 can be ASICs from Alphabet Corporation in Mountain View, CA. Cloud TPU includes a cloud service that makes TPUs available as a scalable resource for processing tasks, such as, but not limited to, machine learning workloads that can run on frameworks such as, but not limited to, TensorFlow, PyTorch, and JAX.

[0171] Chip 2100 can contain any number of TPUs, which can include Tensor Cores 2106. A Tensor Core 2106 can include one or more Core Sequencers 2108, Vector Processing Units (VPUs) 2110, Matrix Multiplication Units (MXUs) 2112(A)-2114(N), where N is an integer greater than 1, and a Transpose Permutation Unit 2116. A Core Sequencer 2108 can retrieve instructions (e.g., VLIW (Very Long Instruction Word)) from the Instruction Memory (Imem) of the Core 2106, perform scalar operations using a scalar data memory (Smem) and scalar registers (Sregs) (not shown), and forward vector instructions to the Vector Processing Unit (VPU) 2110. The instructions can, for example, initiate eight operations: two scalar ALUs, two vector ALUs, vector loading and storage, and a pair of slots that queue data to and from the matrix multiplication and transposition units.The VPU 2110 can perform vector operations using a large on-chip vector memory (Vmem) and vector registers (Vregs). The VPU 2110 can stream data to and from the MXU by decoupling FIFOs. The VPU 2110 can collect and distribute data to the Vmem via data-level parallelism (2D matrix and vector functional units) and instruction-level parallelism (8 operations per instruction). For example, a large two-dimensional matrix multiplier unit (MXU) 2112(A)-2112(N) can use a systolic array to reduce area and energy, plus large, software-controlled on-chip memories instead of caches. The transpose-reduction-permutation unit 2116 can perform matrix transpositions, reductions, and permutations of the VPU 2110 lanes (e.g., 128 × 128). High-bandwidth 2104 memory can be used for on-chip applications. One or more 2100 chips can be interconnected for data processing.For example, one or more Chip 2100s can be connected as a torus, e.g., a 2D torus. Chip 2100s can also include any number (e.g., four) of Inter-Core Interconnect (ICI) links 2118, which can enable direct connections between chips to form a supercomputer.

[0172] With respect to any of the processors in Chip 2100 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., sequentially or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0173] In at least one embodiment, Chip 2100 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0174] Fig. Figure 22 illustrates a vector processor according to at least one embodiment. Vector processor 2200 can support a RISC-V standard. Vector processor 2200 can include one or more cores 2210 (e.g., scalar units) with one or more vector processing units (VPUs) 2242 (e.g., vector units) that can perform, for example, some or all of the operations described above or elsewhere herein. Core 2210 can include Customer Extension (ACE) 2216, which can be used to communicate customer-specific instructions to processor 2200. Core 2210 can include a 1-cycle multiplier and 1-cycle instruction / data local memory (ILM / DLM) for increased parallelism by enabling concurrent instruction fetching and data access.The Memory Management Unit (MMU) 2224 can manage system memory and cache and provide branch execution, instruction-pair output, L1 instruction / data caches, and local memory storage. The Core 2210 can include a Physical Memory Protection Unit (PMP / PPMA) 2222. The Core 2210 can include a Digital Signal Processor (DSP) 2228 and a Floating Point Unit (FPU) 2226, as well as a Load / Store Unit (LSU) 2232 to interface with the memory hierarchy (D$ 2234 and I$ 2230). The Core 2210 can also include a Branch Prediction Unit (BPU) 2218 and a Multiplier Unit (MPU) 2220.

[0175] The Vector Processing Unit (VPU) 2242 can include one or more Vector Functional Units (FUs) 2246(A)-2246(N) which can be chained together for parallel processing, independent memory paths for loading / storing RISC-V vector (RVV) via ACE-RVV 2248 and AND Streaming Port (ASP) 2244 and a Vector Load / Store Unit (VLSU) 2250.

[0176] The Vector Processor 2200 can include bus interfaces such as, among others, an L2 cache memory port 2256 for cacheable access, an MMIO port 2254 for non-cacheable access, an input / output coherence port (IOCP) 2258 for a cacheless bus master, local memory access ports for ILM / DLM 2212 and High-Bandwidth Vector Memory (HVM) 2236 access, and a shared peripheral port (SPP) 2252 for external peripherals. Other memory ports include an LM slave port AXI 2202 and an HVM sub-port AXI 2204.

[0177] With respect to any of the processors in Processor 2200 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0178] In at least one embodiment, Vector Processor 2200 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0179] Fig. Figure 23A illustrates a diagram of an example multi-core tile processor microarchitecture. Multi-core tile processor in Fig. 23 can include a speech processing processor. As in Fig. As illustrated in Figure 23A, each "tile" of the processor architecture is a processing element linked together using a network-on-chip (NoC) that can be used to perform, for example, some or all of the operations described above or elsewhere herein. For example, each tile may include an instruction dispatcher 2304 and an integer (INT) 2306 and floating-point (FP) unit 2308, as well as a load / store unit (LSU) 2312 to interface with the memory hierarchy (data cache (D$) 2310 and instruction cache (I$) 2314) and a network (NET) 2316 interface for communicating with other tiles of the architecture. Some tiles in Processor 2300 may include memory controllers 2302 for managing and controlling memory, as further described herein. Processor 2300 may have a functional slice architecture. The 2300 processor can be located on an application-specific integrated circuit (ASIC), and Fig. 23A can represent the layout of the ASIC. Processor 2300 can include a co-processor designed to execute instructions for a predictive model. The predictive model is any model configured to make a prediction from input data. The predictive model can use a classifier to make a classification prediction. The predictive model can be a machine learning model, such as, but not limited to, a tensor flow model, and Processor 2300 is a tensor streaming processor.

[0180] The 2300 processor can employ different microarchitectures, which vary the functional units in each tile. Fig. 23B, disaggregate. Instead, the function tiles of the 2300 processor can be aggregated into a variety of functional process units (hereafter referred to as "slices") 2304, each corresponding to a specific function type (e.g., FP / INT, NET, MEM). For example, as shown in Fig. As shown in Figure 23B, each slice can correspond to a column of functional tiles extending in a north-south direction. Additionally, the processor includes communication traces to carry data between the tiles of different slices, each running horizontally in an east-west direction. Each communication trace can be connected to any of the slices 2304 of the 2300 processor.

[0181] The 2304 slices of the 2300 processor can each correspond to a different function and can include arithmetic logic slices (e.g., FP / INT), track-change slices (e.g., NET), and memory slices (e.g., MEM). The arithmetic logic units perform one or more arithmetic and / or logical operations on the data received via the communication lanes to produce output data. Examples of arithmetic logic units include matrix multiplication units and vector multiplication units. The memory slices contain memory cells that store data. The memory slices can provide data to other slices via the communication lanes. The memory slices can also receive data from other slices via the communication lanes. The track-change slices can configurably route data from one communication lane to any other communication lane.For example, data from a first track can be provided to a second track via a track-change slice. In some embodiments, the track-change slice can be implemented as a crossbar switch. Each 2304 slice also includes its own instruction queue (not shown), which stores instructions, and an instruction control unit (ICU) to control instruction execution. The instructions in a given instruction queue can only be executed by tiling within their associated functional slice and cannot be executed by the processor's other slice.

[0182] By arranging the tiles of the 2300 processor into different functional slices 2304, the on-chip instruction and control flow of the 2300 processor can be decoupled from the data flow. For example, an arrow in Fig. 23B the flow of instructions within the processor architecture according to some embodiments. Another arrow in Fig. Figure 23B illustrates data flow within the processor architecture according to at least one embodiment. As shown, instructions and control flow in a first direction across the tiles of the 2300 processor (e.g., north-south, along the length of the functional slices, as indicated by the first arrow), while data flows in a second direction across the tiles of the 2300 processor (e.g., east-west, across the functional slices, as indicated by the second arrow), which is perpendicular to the first direction.

[0183] Different functional slices of the processor can correspond to MEM (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numeric interpretation module), and SXM (switching and permutation module). Each slice can contain N tiles, all of which can be controlled by the same instruction control unit (ICU) (not shown). Each tile of the processor can correspond to an execution unit organized as an xM SIMD tile. For example, each tile of the processor's on-chip memory can be organized to atomically store an L-element vector. Thus, a MEM slice with N tiles can work together to store or process a large vector (e.g., with a total of N × M elements).

[0184] Tiles within the same slice can execute instructions in a staggered manner, with instructions being issued tile by tile within the slice over a period of N cycles. Functional slices can be physically arranged on-chip to enable efficient data flow for pipelined execution over hundreds of cycles for common patterns. Data flows can perform a single "U-turn" (direction change), corresponding to a single matrix operation, before being written back to memory. In some embodiments, a given data flow can change direction multiple times (due to multiple matrix and vector operations) before the resulting data is written back to memory.

[0185] To achieve good single-threaded performance, a conventional multi-core processor design (e.g., as in Fig. Figure 23A illustrates this. Typically, a significant portion of the silicon area is dedicated to exposing and utilizing instruction-level parallelism (ILP). This usually involves register renaming schemes and large instruction windows, through which the instructions have no explicit understanding of the hardware on which they are executed, while maintaining the illusion of in-order program execution. In contrast, when using a processor (e.g., TSP) with a functional slice architecture, the TSP compiler generates an explicit plan for how the processor will execute the microprogram. The compiler specifies when each operation will be performed, which functional slices will carry out the work, and which STREAM registers will hold the operands. The compiler maintains a highly reliable (cycle-accurate) model of the TSP's hardware state, enabling the microprogram to orchestrate the data flow.

[0186] The Processor 2300 (e.g., TSP) can use a web-hosted compiler that takes a model (e.g., a machine learning model, such as, but not limited to, a TensorFlow model) as its input and outputs a proprietary instruction stream targeting the Processor-TSP hardware. The compiler is responsible for coordinating the program's control and data flow and specifies any instruction-level parallelism by explicitly bundling instructions that can and should be executed concurrently so they can be sent together. The primary hardware structure is the architecturally visible streaming register file (STREAMs), described in more detail below, which serves as the conduit through which operands flow from memory slices (e.g., SRAM) to functional slices and vice versa.

[0187] The processor's MEM unit serves as: (1) storage for model parameters, microprograms, and the data on which they operate, and (2) a network-on-chip (NoC) for communicating data operands from MEM to the functional slices and computed results back to MEM. In some embodiments, the on-chip memory occupies approximately 75% of the processor's chip area. In some embodiments, due to the processor's bandwidth requirements, the on-chip memory of the MEM tiles may be SRAM rather than DRAM. The processor's on-chip memory capacity determines (i) the number of ML models that can reside on the chip simultaneously, (ii) the size of any given model, and (iii) the partitioning of large models to fit into multi-chip systems. In some embodiments, the processor's MEM system provides a plurality of memory slices arranged in two distinct hemispheres (referred to as "MEM WEST" and "MEM 3").“MEM EAST” is the name given to these organizations.

[0188] The memory slices of each hemisphere can be mirrored so that the slices can be physically numbered {0,...}. In at least one embodiment, NEF 2116 can also have a memory slice 0 {L} in the east hemisphere 410 and {L,... 0} in the west hemisphere 405, such that memory slice 0 for each hemisphere corresponds to the slice that is closest to the VXM slices 415 between the hemispheres, with each hemisphere having L slices. The direction of data transmission toward the center of the chip can be described as inward, while data transmission toward the outermost (easternmost or westernmost) edge of the chip can be described as outward. Although the hemispheres of the processor's memory can be described as east and west, it is understood that in other embodiments other names can be used to refer to the different hemispheres of memory.

[0189] In some embodiments, a streaming register file, referred to as STREAMS, transfers operands and results between the SRAM of the MEM slices and the functional slices of the processor. In some embodiments, a large number of MEM slices (e.g., between 2 and 10 adjacent MEM slices) can be physically organized as a set. Each set of slices can be located between a pair of STREAM register files, so that each slice is able to read from or write to the STREAM registers in both directions. Placing STREAM register files between sets of MEM slices reduces the number of cycles required for data operands to be transferred across one hemisphere (e.g., by a factor equal to the number of slices per set). The number of slices per set can be configured based on a distance over which data can be transferred in a single clock cycle.

[0190] With regard to any one of the processors in Fig. 23 and any components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions including API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory). A result of API(s) may then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents.

[0191] In at least one embodiment, processor 2300 may include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or to otherwise perform any of the operations described above or elsewhere herein. Software designs

[0192] The following figures present, without limitation, examples of software constructs for implementing at least one embodiment.

[0193] Fig. Figure 24 illustrates a software stack of a programming platform according to at least one embodiment. A programming platform may include a platform for utilizing hardware on a computing system to accelerate computational tasks. A programming platform may be accessible to software developers through libraries, compiler directives, and / or extensions of programming languages ​​in at least one embodiment. A programming platform may be CUDA, Radeon Open Compute Platform (“ROCm”), OpenCL (OpenCL™ is developed by the Khronos group), SYCL, or Intel OneAPI.

[0194] A software stack 2400 of a programming platform can provide an execution environment for an application 2401. The application 2401 can include any computer software capable of running on the software stack 2400. The application 2401 can include an artificial intelligence (“AI”) / machine learning (“ML”) application, a high-performance computing (“HPC”) application, a virtual desktop infrastructure (“VDI”), or a data center workload.

[0195] Application 2401 and software stack 2400 run on hardware 2408. Hardware 2408 can include one or more GPUs, CPUs, FPGAs, AI engines, and / or other types of computing devices that support a programming platform. Software stack 2400 can be vendor-specific and compatible only with devices from a single vendor or vendors, such as CUDA, ROCm, OneAPI, OpenCL, or other implementations. Hardware 2408 can include a host connected to one or more devices that can be accessed to perform computing tasks via application programming interface (API) calls.A device within Hardware 2408 can, in at least one embodiment, include a GPU, FPGA, AI engine, or other computing device (but may also include a CPU) and its memory, as opposed to a host within Hardware 2408, which can include a CPU (but may also include a computing device) and its memory. With respect to any of the Hardware 2408 described above or elsewhere herein, one or more of the APIs described herein can, for example, be compiled into instructions that can be fetched by instruction-fetch logic, decoded by a processor decoder, scheduled for execution by a scheduler (e.g., in sequence or out of sequence), executed by execution logic, reordered, and then shut down by shutdown logic. API(s) (and / or compiled instructions including API(s)) can be stored in any memory outside or inside the processor (e.g.,The result of an API (or APIs) can be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. One or more of the APIs described herein may include a call. One or more of the APIs described herein may include a library or a portion of a library to perform a function described by the call. One or more of the APIs described herein may include a call and a library or a portion of a library to perform a function described by the call.

[0196] A software stack 2400 of a programming platform can include a number of libraries 2403, a runtime 2405, an optional driver / interface 2407, and a device kernel driver 2408. Each of the libraries 2403 can contain data and programming code that can be used by computer programs and during software development. Libraries 2403 can include previously written code and subroutines, classes, values, type specifications, configuration data, documentation, auxiliary data, and / or message templates. Libraries 2403 can include functions that can be optimized for execution on one or more device types. Libraries 2403 can include functions for performing mathematical, deep learning, and / or other types of operations on devices.Libraries 2403 can be associated with corresponding APIs 2402, which may include one or more APIs that expose functions implemented in libraries 2403. A processor (e.g., CPU, GPU) can execute, call, or otherwise use one or more APIs to prioritize cores. For example, a first kernel (e.g., parent kernel) can start a second kernel (e.g., child kernel), and the second kernel can be used by a processor to start additional kernels (e.g., grandchildren kernels) independently of the first kernel. A processor can execute an API or call an API from the executing memory to support dynamic stream priority (e.g., updating the priority while a stream is being used to perform operations).For example, when a processor performs the API, it allows a programmer to copy the current priority from one current to one or more other currents.

[0197] Software Stack 2400 can include an API to support dynamic stream priority (e.g., updating the priority while a stream is being used to perform operations), allowing a programmer to set a stream's priority at any time after its creation. Software Stack 2400 can also include an API to support dynamic stream priority (e.g., updating the priority while the stream is being used to perform operations), allowing a programmer to retrieve a stream's current priority, where the priority is one of several attributes of the stream. Finally, Software Stack 2400 can include an API to support dynamic stream priority (e.g., updating the priority while the stream is being used to perform operations), allowing a programmer to retrieve a stream's current priority as a single attribute.Software stack 2400 can include an API to support dynamic stream priority (e.g., updating the priority while the stream is being used to perform operations), allowing a programmer to start a kernel to perform operations on a stream with a set priority that may differ from the stream priority. Software stack 2400 can also include an API to specify whether an object (e.g., a thread synchronization object, such as, but not limited to, a barrier) is tracking whether all data movement operations for a set of threads running on a GPU can be completed, has a specified state after a specified period, where a specified state can be a state indicating that data has been moved and is ready for use, and is specified using an expected parity value as input to the API.

[0198] Software Stack 2400 can include one or more APIs for updated kernels. A processor can execute an API or call an API from the executable memory to update an existing API and support context-free kernels, allowing a programmer to add a kernel node to a graph without a graphics context, so that a graphics context can be dynamically associated with a kernel at runtime. Software Stack 2400 can include one or more APIs to allow a programmer to receive a kernel identifier and a graphics context as separate parameters from a kernel node, so that parameters can be obtained from kernels and from context-free kernels. Software Stack 2400 can include one or more APIs to use parallel processor(s), such as, but not limited to, one or more graphics processing units (GPUs), to process task graphs (e.g.,to start task graphs) and execute one or more task graphs (e.g., including one or more programs).

[0199] Software Stack 2400 can include one or more APIs to associate one or more instructions with one or more memory order operations, such as, but not limited to, a fence or membar operation. Instructions can be associated with one or more domains, so that a memory order operation is executed in association with one or more specific domains without interfering with instructions in other domains. An API can indicate that a thread has arrived (for example, at a thread synchronization barrier) or has completed a stage of work with respect to asynchronous data movement operations on a GPU.Software stack 2400 can include one or more to allow programmers to manually specify an expected transaction count when a thread has finished a work stage, which can be used to update an object that tracks whether all data movement operations for a set of threads can be completed.

[0200] Application 2401 can be written as source code that is compiled into executable code, as shown below in conjunction with Fig. 25 and Fig. Section 26 discusses this in more detail. Executable code of Application 2401 can run, at least partially, on an execution environment provided by Software Stack 2400. During the execution of Application 2401, code may be encountered that needs to run on a device, as opposed to a host. In such a case, Runtime 2405 can be invoked to load and start the necessary code on the device. Runtime 2405 can include any technically feasible runtime system capable of supporting the execution of Application 2401.

[0201] Runtime 2405 can be implemented as one or more runtime libraries associated with corresponding APIs, shown as API 2404 and API 2404, respectively. One or more such runtime libraries may include, among other things, functions for memory management, execution control, device management, error handling, and / or synchronization. Memory management functions may include functions for allocating, unallocating, and copying device memory, as well as for transferring data between host memory and device memory. Execution control functions may include functions for starting a function (sometimes called a "kernel" if a function is a global function callable from a host) on a device and setting attribute values ​​in a buffer maintained by a runtime library for a given function to be executed on a device.

[0202] Runtime libraries and their corresponding APIs can be implemented in any technically feasible way. One (or any number of) APIs can expose a low-level set of functions for fine-grained control of a device, while another (or any number of) APIs can expose a higher-level set of such functions. A high-level runtime API can be built upon a low-level API. One or more runtime APIs can be language-specific APIs that can be layered on top of a language-independent runtime API.

[0203] An optional driver or interface 2407 can be implemented, for example, for CUDA and ROCm implementations, which are described below. Optional driver / interface 2407 can be associated with an optional driver or interface API(s), such as, but not limited to, CUDA and / or ROCm API(s).

[0204] One or more processors disclosed in "Processing Systems" can execute, access, or otherwise use Software Stack 2400. For example, System-on-a-Chip 1100, Parallel Processor 1200, Graphics Multiprocessor 1234, Processor 1300, Processor 1400, Accelerator 1500, Neuromorphic Processor 1605, Supercomputer 1700, Acceleration Processing Unit 1800, Processor 1900, Processor 2000, Tensor Processing Unit 2100, Processor 2200, and Speech Processing Unit 2300 can execute, use, call, or otherwise implement (for example, by accessing memory) one or more APIs included in Software Stack 2400.

[0205] Device kernel driver 2408 can be configured to facilitate communication with an underlying device. Device kernel driver 2408 can provide low-level functionality upon which APIs, such as API 2404 or APIs 2404, and / or other software depend. Device kernel driver 2408 can be configured to compile intermediate representation ("IR") code into binary at runtime. For CUDA or other implementations, such as, but not limited to, ROCm, OneAPI, or OpenCL, device kernel driver 2408 can compile parallel-threaded execution ("PTX") IR code, which is not hardware-specific, into binary for a specific target device at runtime (with caching of compiled binary), sometimes referred to as "finalizing" code. This can allow finalized code to run on a target device that may not have existed when the source code was originally compiled into PTX code.Alternatively, device source code can be compiled offline into binary code, without the need for the device kernel driver to compile 2408 IR code at runtime.

[0206] Processors described elsewhere herein, such as, but not limited to, processors in Fig. 11- Fig. 23, may include one or more circuits to cause one or more tables to map one or more tokens, generated by one or more neural networks, into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software, e.g.Software stack 2400 to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0207] According to at least one embodiment, software stack 2400 of Fig. 24 in a CUDA implementation. A CUDA software stack 2400, on which an application 2401 can be started, can include CUDA libraries 2403, a CUDA runtime 2405, a CUDA driver 2407, and a device kernel driver 2408. CUDA software stack 2400 can be run on hardware 2809, which can include a GPU that supports CUDA and is developed by NVIDIA Corporation of Santa Clara, CA.

[0208] Application 2401, CUDA runtime 2405, and device kernel driver 2408 can perform functionalities described above and elsewhere herein. CUDA driver 2407 can include a library (libcuda.so) that can implement a CUDA driver API 2406. Similar to a CUDA runtime API 2404 implemented by a CUDA runtime library (cudart), CUDA driver API 2406 can expose, among other things, functions for memory management, execution control, device management, error handling, synchronization, and / or graphics interoperability. CUDA driver API 2406 may differ from CUDA runtime API 2404 in that CUDA runtime API 2404 simplifies device code management by providing implicit initialization, context (analogous to a process) management, and module management (analogous to dynamically loaded libraries).In contrast to the high-level CUDA runtime API 2404, the CUDA driver API 2406 can be a low-level API that provides more granular control of the device, particularly regarding contexts and module loading. The CUDA driver API 2406 can expose context management functions that cannot be exposed by the CUDA runtime API 2404. The CUDA driver API 2406 can also be language-independent and support languages ​​such as OpenCL, in addition to the features of the CUDA runtime API 2404. Furthermore, development libraries, including the CUDA runtime API 2405, can be considered separate from driver components, including the user-mode CUDA driver 2407 and the kernel-mode device driver 2408 (sometimes referred to as the "display" driver).

[0209] CUDA libraries 2403 can include mathematical libraries, deep learning libraries, parallel algorithm libraries, and / or signal / image / video processing libraries that can be used by parallel computing applications, such as application 2401. CUDA libraries 2403 can include, but are not limited to, mathematical libraries such as a cuBLAS library, which is an implementation of Basic Linear Algebra Subprograms (“BLAS”) for performing linear algebra operations, a cuFFT library for calculating Fast Fourier Transforms (“FFTs”), and a cuRAND library for generating random numbers. CUDA libraries 2403 can include, but are not limited to, deep learning libraries such as a cuDNN library of primitives for deep neural networks and a TensorRT platform for high-performance deep learning inference.

[0210] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 11- Fig. 23, include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software, e.g.Software stack 2400 to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0211] According to at least one embodiment, software stack 2400 of Fig. 24 in an ROCm implementation. An ROCm software stack 2400, on which an application 2401 can be started, includes a language runtime 2403, a system runtime 2405, a thunk 2407, and an ROCm kernel driver 2408. The ROCm software stack 2400 runs on hardware 2409, which may include a GPU that supports ROCm and is developed by AMD Corporation of Santa Clara, CA.

[0212] Application 2401 can perform similar functionalities as described above in conjunction with Fig. 24 discussed. Additionally, language runtime 2403 and system runtime 2405 can perform similar functionalities to runtime 2405, which are discussed above in conjunction with Fig. 24 is discussed. Language Runtime 2403 and System Runtime 2405 can differ in that System Runtime 2405 is a language-independent runtime that implements an ROCr System Runtime API 2404 and makes use of a Heterogeneous System Architecture (“HSA”) runtime API. The HSA runtime API can include, among other things, a thin user-mode API that exposes interfaces for accessing and interacting with an AMD GPU, including functions for memory management, execution control via architected kernel dispatch, error handling, system and agent information, and runtime initialization and shutdown. In contrast to System Runtime 2405, Language Runtime 2403 can be an implementation of a language-specific runtime API 2402 layered on top of an ROCr System Runtime API 2404.A language runtime API can include, among other things, a Heterogeneous Compute Interface for Portability ("HIP") language runtime API, a Heterogeneous Compute Compiler ("HCC") language runtime API, or an OpenCL API. HIP, in particular, is an extension of the C++ programming language with functionally similar versions of CUDA mechanisms, and a HIP language runtime API can include features similar to those of the CUDA runtime API described above. Fig. 24 is discussed, including but not limited to functions for memory management, execution control, device management, error handling and synchronization.

[0213] Thunk (ROCt) 2407 can be an interface 2406 that can be used to interact with the underlying ROCm driver 2408. The ROCm driver 2408 can be a ROCK driver that is a combination of an AMDGPU driver and an HSA kernel driver (amdkfd). The AMDGPU driver can be a device kernel driver for GPUs, developed by AMD, that performs similar functionalities to the device kernel driver 2409, which is mentioned above in conjunction with... Fig. 24 is discussed. An HSA kernel driver can be a driver that allows different types of processors to share system resources more effectively via hardware features.

[0214] Various libraries (not shown) can be included in the ROCm software stack 2400 via language runtime 2403 and provide functionality similar to CUDA libraries 2403, which are mentioned above in conjunction with Fig. 24 is discussed. Various libraries may include mathematical, deep learning and / or other libraries, such as, but are not limited to, a hipBLAS library that implements functions similar to those of CUDA cuBLAS, a rocFFT library for calculating FFTs, and a rocFFT library for calculating FFTs, similar to CUDA cuFFT.

[0215] Processors described elsewhere herein, such as, but not limited to, processors in Fig. 11- Fig. 23, may include one or more circuits to cause one or more tables to map one or more tokens, generated by one or more neural networks, into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software, e.g.Software stack 2400 to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0216] According to at least one embodiment, software stack 2400 of Fig. 24 in an OpenCL implementation. An OpenCL software stack 2400, on which an application 2401 can be started, may include an OpenCL framework 2403, an OpenCL runtime 2405, and a driver 2408. The OpenCL software stack 2400 can run on hardware 2409 that is not vendor-specific. Because OpenCL is supported by devices developed by different vendors, specific OpenCL drivers may be required to work compatiblely with hardware from such vendors.

[0217] Application 2401, OpenCL runtime 2405, device kernel driver 2408, and hardware 2409 can perform similar functionalities to other implementations of Application 2401, runtime 2405, device kernel driver 2408, and hardware 2409, respectively, which are described above in conjunction with Fig. 24 are discussed. Application 2401 may further include an OpenCL kernel (not shown) with code to be executed on a device.

[0218] OpenCL can define a "platform" that allows a host to control devices connected to it. An OpenCL framework can provide a platform layer API and a runtime API, shown as Platform API 2402 and Runtime API 2404. Runtime API 2404 can use contexts to manage kernel execution on devices. Each identified device can be associated with a specific context, which Runtime API 2404 can use to manage command queues, program objects, and kernel objects, and to share memory objects, among other things, for that device. Platform API 2402 can expose functions that allow the use of device contexts to select and initialize devices, submit work to devices via command queues, and enable data transfer to and from devices.Additionally, the OpenCL framework can provide various built-in functions (not shown), including mathematical functions, relational functions, and image processing functions, among others.

[0219] A compiler (not shown) can also be included in OpenCL framework 2403. Source code can be compiled offline before an application is run, or online during application execution. Unlike CUDA and ROCm, OpenCL applications can be compiled online by a compiler that is representative of any number of compilers that can be used to compile source code and / or IR code, such as, but not limited to, Standard Portable Intermediate Representation ("SPIR-V") code, into binary code. Alternatively, OpenCL applications can be compiled offline before such applications are executed.

[0220] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 11- Fig. 23, include one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software, e.g.Software stack 2400 to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens that are mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0221] According to at least one embodiment, software can be supported by a programming platform configured to support various programming models, middleware, and / or libraries and frameworks upon which an application may rely. The application may be an AI / ML application implemented, for example, using a deep learning framework such as, but not limited to, MXNet, PyTorch, or TensorFlow, which may rely on libraries such as cuDNN, NVIDIA Collective Communications Library (“NCCL”), and / or NVIDIA Developer Data Loading Library (“DALI”) CUDA libraries to provide accelerated computing on underlying hardware.

[0222] The programming platform can be one of a CUDA, ROCm, or OpenCL platform, as described above in conjunction with Fig. 24. The programming platform can support multiple programming models, which can be abstractions of an underlying computer system that allow expressions of algorithms and data structures. Programming models can expose features of underlying hardware to improve performance. Programming models can include CUDA, HIP, OpenCL, C++ Accelerated Massive Parallelism (“C++AMP”), Open Multi-Processing (“OpenMP”), Open Accelerators (“OpenACC”), and / or Vulcan Compute.

[0223] Libraries and / or middleware can provide implementations of programming model abstractions. Such libraries can include data and code that can be used by computer programs and during software development. Middleware can include software that provides services for applications beyond those available from the programming platform. Libraries and / or middleware can include cuBLAS, cuFFT, cuRAND, and other CUDA libraries, or rocBLAS, rocFFT, rocRAND, and other ROCm libraries.Additionally, libraries and / or middleware may include NCCL and ROCm Communication Collectives Library (“RCCL”) libraries, which provide communication routines for GPUs, an MIOpen library for deep learning acceleration, and / or a proprietary library for linear algebra, matrix and vector operations, geometric transformations, numerical solvers, and related algorithms.

[0224] Application frameworks can depend on libraries and / or middleware. Any application framework can be a software framework used to implement a standard application software structure. Returning to the AI / ML example discussed above, an AI / ML application can be implemented using a framework such as, but not limited to, Caffe, Caffe2, TensorFlow, Keras, PyTorch, or MxNet deep learning frameworks.

[0225] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 11- Fig. 23, include one or more circuits to cause one or more tables to map one or more tokens, generated by one or more neural networks, into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be implemented by software, e.g.,The programming platforms described herein may be configured to cause one or more tables to map one or more tokens generated by one or more neural networks to one or more tokens to be used by one or more application programming interfaces, to run one or more neural networks that generate tokens mapped using the tables, or otherwise to perform any of the operations described above or elsewhere herein.

[0226] Fig. Figure 25 illustrates compiling code for execution on one of the programming platforms described above. Fig. 24 according to at least one embodiment. A compiler 2501 is configured to receive source code 2500, compile source code 2500, and output an executable file 2510. Compiler 2501 can be configured to convert source code 2500 into host executable code 2507 for execution on a host and device executable code 2508 for execution on a device. Source code 2500 can be compiled either offline before an application is executed or online during the execution of an application. Source code 2500 can contain code in any programming language supported by compiler 2501, such as, but not limited to, C++, C, Fortran, etc. Source code 2500 can be contained in a single source file, with a mixture of host code and device code, the positions of which are indicated therein. A single source file can be a .cu file containing CUDA code, or a .hip file.A .cpp file containing HIP code, or a file in another format containing both host code and device code. Alternatively, Source Code 2500 may contain multiple source code files instead of a single source file, in which host code and device code may be separated. Compiler 2501 includes or has access to one or more libraries to recognize a sequence of API calls to execute a single fused API, where a single fused API is a combined API for two or more APIs. In at least one embodiment, Compiler 2501 may be an NVIDIA CUDA compiler (“NVCC”) for compiling CUDA code into .cu files, an HCC compiler for compiling HIP code into .hip.cpp files, or other compilers.

[0227] Compiler 2501 can be configured to compile source code 2500 into host executable code 2507 for execution on a host and device executable code 2508 for execution on a device. Compiler 2501 performs operations including parsing source code 2500 into an abstract system tree (AST), performing optimizations, and generating executable code. If source code 2500 contains a single source file, Compiler 2501 can separate device code from host code in such a single source file, compile device code and host code into device executable code 2508 and host executable code 2507 respectively, and join the device executable code 2508 and host executable code 2507 together in a single file.

[0228] Compiler 2501 can include a compiler frontend 2502, a host compiler 2505, a device compiler 2506, and a linker 2509. Compiler frontend 2502 can be configured to separate device code 2504 from host code 2503 into source code 2500. Device code 2504 can be compiled by device compiler 2506 into executable device code 2508, which, as described, can include binary code or IR code, in at least one embodiment. Separately, host code 2503 can be compiled by host compiler 2505 into executable host code 2507. For NVCC other compilers, such as, but not limited to, those for oneAPI, ROCm and OpenCL, Host Compiler 2505 can be a universal C / C++ compiler that outputs native object code, while Device Compiler 2506 can be a Low Level Virtual Machine (“LLVM”)-based compiler that forks an LLVM compiler infrastructure and outputs PTX code or binary code.For HCC, both host compiler 2505 and device compiler 2506 can be LLVM-based compilers that output binary target code.

[0229] After compiling source code 2500 into executable host code 2507 and executable device code 2508, linker 2509 can link executable host code 2507 and device code 2508 together in an executable file 2510. Native object code for a host and PTX or binary code for a device can be linked together in an Executable and Linkable Format (“ELF”) file, which is a container format used to store object code. Executable host code 2507 and executable device code 2508 can be in any suitable format, such as, but not limited to, binary code and / or IR code. In the case of CUDA, executable host code 2507 can contain native object code, and executable device code 2508 can contain code in PTX intermediate representation, in at least one embodiment. In the case of ROCm, both executable host code 2507 and executable device code 2508 can contain binary target code, in at least one embodiment.Other implementations, such as, but not limited to, oneAPI, OpenCL, are being considered and can be implemented similarly to the CUDA and ROCm implementations above.

[0230] Source code 2500 can be translated before source code is compiled. Source code is passed through a translation tool (not shown) that translates source code 2500 into translated source code. A compiler 2501 can be used to compile translated source code into executable host code 2507 and executable device code 2508 in a process similar to the compilation of source code 2500 into executable host code 2507 and executable device code 2508 by compiler 2501, as described above in conjunction with Fig. 25 discussed.

[0231] A translation performed by a translation tool can be used to port Source Code 2500 for execution in a different environment than the one in which it was originally intended to run. The translation tool may include a HIP translator, which is used to "translate" CUDA code intended for a CUDA platform into HIP code that can be compiled and executed on a ROCm platform. Source Code 2500 translation may involve parsing Source Code 2500 and converting calls to APIs provided by one programming model (e.g., CUDA) into corresponding calls to APIs provided by another programming model (e.g., HIP), as shown below in conjunction with... Fig. 26 is discussed in more detail. Returning to the example of translating CUDA code, calls to the CUDA runtime API, CUDA driver API, and / or CUDA libraries can be converted into corresponding HIP API calls. Automated translations performed by translation tool 2501 can sometimes be incomplete, requiring additional manual effort to fully port source code 2500.

[0232] One or more of the techniques described herein may utilize other methods for converting one code type to another to enable interchangeability between different device architectures. In at least one embodiment, an application for one platform (e.g., a CUDA application) may be compiled into code for implementation on another platform (e.g., an AMD processor, an Intel processor, or another processor). For example, Source Code 2500 may contain source code for one platform (e.g., CUDA). Compiler 2501 may compile Source Code 2500 into an executable file 2510 that can be used by another platform (e.g., AMD or Intel). Programming toolkits may enable applications for one platform (e.g., CUDA) to be compiled (e.g., natively) for another platform (e.g., AMD or Intel).For example, a GPGPU programming toolkit can enable CUDA applications to be compiled natively for AMD GPUs. Programs (e.g., CUDA programs) or their architecture do not need to be modified or translated into another language before being compiled to code for a different platform. A compiler can accept the same command-line options and programming dialect (e.g., CUDA dialect) as another compiler (e.g., nvcc for CUDA), acting as a drop-in replacement to mimic the installation of a toolkit (e.g., NVIDIA CUDA Toolkit), allowing existing architecture tools and scripts (e.g., such as CMake) to function without further modification. In at least one embodiment, an nvcc-compatible compiler can be used to compile nvcc-compatible CUDA for AMD GPUs, including PTX assembly. Implementations of CUDA runtime and driver APIs for AMD GPUs can be used. Libraries (e.g.,Open-source wrapper libraries can provide APIs, such as CUDA-X APIs, by delegating to the corresponding ROCm libraries. One exemplary implementation is SCALE from Spectral Compute in London, England. Rather than providing a new way to write GPGPU software, SCALE allows programs written using the widely adopted CUDA language to be compiled directly for AMD GPUs. Additional implementations may include a Clang compiler, which provides a language frontend and tool infrastructure for languages ​​in the C language family (C, C++, Objective C / C++, OpenCL, CUDA, and RenderScript). In at least one embodiment, compilers described herein, such as, but not limited to, Compiler 2501, Compiler 2505, and / or Compiler 2506, may include one or more circuits to compile code (e.g.,to compile (CUDA, HIP, OpenCL, OneAPI or others) to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces to run one or more neural networks that generate tokens mapped using the tables, and / or to perform any of the operations described above or elsewhere herein.

[0233] Fig. Figure 26 illustrates a System 2600 configured to compile and execute CUDA source code 2610 using various types of processing units according to at least one embodiment. System 2600 includes CUDA source code 2610, a CUDA compiler 2650, host executable code 2670(1), host executable code 2670(2), CUDA device executable code 2684, a CPU 2690, a CUDA-enabled GPU 2694, a GPU 2692, a CUDA-to-HIP translation tool 2620, HIP source code 2630, a HIP compiler driver 2640, an HCC 2660, and HCC device executable code 2682.

[0234] CUDA source code 2610 can be a collection of human-readable code written in a CUDA programming language. A CUDA programming language can be an extension of the C++ programming language that includes mechanisms to define device code and distinguish between device code and host code. Device code can include source code that, after compilation, can be executed in parallel on a device. A device can be a processor optimized for parallel instruction processing, such as, among others, the CUDA-enabled GPU 2690, GPU 2692, or another GPGPU. Host code is source code that, after compilation, can be executed on a host. A host is a processor optimized for sequential instruction processing, such as, among others, the CPU 2690.

[0235] CUDA source code 2610 can contain any number (including zero) of global functions 2612, any number (including zero) of device functions 2614, any number (including zero) of host functions 2616, and any number (including zero) of host / device functions 2618. Global functions 2612, device functions 2614, host functions 2616, and host / device functions 2618 can be mixed in CUDA source code 2610. Each of the global functions 2612 can be executable on a device and callable by a host. One or more of the global functions 2612 can therefore act as entry points to a device. Each of the global functions 2612 can be a kernel. In a technique known as dynamic parallelism, one or more of the global functions 2612 can define a kernel that is executable on and invoked by a device.A kernel can be executed N times in parallel by N different threads on a device during execution (where N is a positive integer).

[0236] Each of the device functions 2614 can be executed on a device and can only be called by such a device. Each of the host functions 2616 can be executed on a host and can only be called by such a host. Each of the host / device functions 2616 can define both a host version of a function that is executable on a host and can only be called by such a host, and a device version of the function that is executable on a device and can only be called by such a device.

[0237] CUDA source code 2610 can also include any number of calls to any number of functions that can be defined via a CUDA runtime API 2602. A CUDA runtime API 2602 can include any number of functions that run on a host to allocate and unallocate device memory, transfer data between host memory and device memory, manage systems with multiple devices, and so on. CUDA source code 2610 can also include any number of calls to any number of functions that can be specified in any number of other CUDA APIs. A CUDA API can be any API designed for use by CUDA code. CUDA APIs can include the CUDA runtime API 2602, a CUDA driver API, APIs for any number of CUDA libraries, and so on, including any API(s) described elsewhere herein.Regarding the CUDA runtime API 2602, a CUDA driver API can be a lower-level API, but it can provide more granular control of a device. Examples of CUDA libraries include cuBLAS, cuFFT, cuRAND, cuDNN, etc.

[0238] CUDA Compiler 2650 can compile input CUDA code (e.g., CUDA source code 2610) to produce executable host code 2670(1) and executable CUDA device code 2684. CUDA Compiler 2650 can be, but is not limited to, NVCC. Executable host code 2670(1) can be a compiled version of host code contained in input source code that is executable on CPU 2690. CPU 2690 can be any processor optimized for sequential instruction processing.

[0239] CUDA Device Executable Code 2684 can be a compiled version of device code contained in submitted source code that is executable on CUDA-enabled GPU 2694. CUDA Device Executable Code 2684 can include binary code. CUDA Device Executable Code 2684 can include IR code, such as, but not limited to, PTX code, which is further compiled at runtime into binary code for a specific target device (e.g., CUDA-enabled GPU 2694) by a device driver. CUDA-enabled GPU 2694 can include any processor optimized for parallel instruction processing that supports CUDA. CUDA-enabled GPU 2694 can be developed by NVIDIA Corporation of Santa Clara, CA.

[0240] The CUDA-to-HIP translation tool 2620 can be configured to translate CUDA source code 2610 into functionally similar HIP source code 2630. HIP source code 2630 can contain a collection of human-readable code in a HIP programming language. HIP code can contain human-readable code in a HIP programming language. A HIP programming language can include an extension of the C++ programming language that includes functionally similar versions of CUDA mechanisms for defining device code and distinguishing between device code and host code. A HIP programming language can include a subset of the functionality of a CUDA programming language. For example, a HIP programming language includes mechanism(s) for defining global functions 2612, but such a HIP programming language may lack support for dynamic concurrency, and therefore global functions 2612 defined in HIP code may only be callable by one host.

[0241] HIP source code 2630 can include any number (including zero) of global functions 2612, any number (including zero) of device functions 2614, any number (including zero) of host functions 2616, and any number (including zero) of host / device functions 2618. HIP source code 2630 can also include any number of calls to any number of functions that can be specified in a HIP runtime API 2632. HIP runtime API 2632 can include functionally similar versions of a subset of functions included in CUDA runtime API 2602. HIP source code 2630 can also include any number of calls to any number of functions that can be specified in any number of other HIP APIs. A HIP API can be any API designed for use by HIP code and / or ROCm.HIP APIs can include the HIP runtime API 2632, a HIP driver API, APIs for any number of HIP libraries, APIs for any number of ROCm libraries, etc.

[0242] The CUDA-to-HIP translation tool 2620 can convert any kernel call in CUDA code from CUDA syntax to HIP syntax and can convert any number of other CUDA calls in CUDA code into any number of other functionally similar HIP calls. A CUDA call can involve a call to a function specified in a CUDA API, and a HIP call can involve a call to a function specified in a HIP API. The CUDA-to-HIP translation tool 2620 can convert any number of calls to functions specified in CUDA Runtime API 2602 into any number of calls to functions specified in HIP Runtime API 2632.

[0243] CUDA-to-HIP translation tool 2620 may include a tool known as Translate-Perl, which performs a text-based translation process. CUDA-to-HIP translation tool 2620 may also include a tool known as Translate-Clang, which performs a more complex and robust translation process compared to Translate-Perl. This process involves parsing CUDA code using Clang (a compiler frontend) and then translating the resulting symbols. Converting CUDA code to HIP code may involve modifications (e.g., manual edits) in addition to those performed by CUDA-to-HIP translation tool 2620.

[0244] The HIP compiler driver 2640 can include a front end that determines a target device 2646 and then configures a compiler compatible with target device 2646 to compile HIP source code 2630. Target device 2646 can include a processor optimized for parallel instruction processing. The HIP compiler driver 2640 can determine target device 2646 in any technically feasible way.

[0245] If target device 2646 is CUDA-compatible (e.g., CUDA-enabled GPU 2694), then HIP compiler driver 2640 can generate a HIP / NVCC compilation command 2642. HIP / NVCC compilation command 2642 can configure CUDA compiler 2650 to compile HIP source code 2630 using a HIP-to-CUDA translation header and a CUDA runtime library. In response to HIP / NVCC compilation command 2642, CUDA compiler 2650 can generate host executable code 2670(1) and CUDA device executable code 2684.

[0246] If target device 2646 is not CUDA-compatible, then HIP compiler driver 2640 can generate a HIP / HCC compilation command 2644. HIP / HCC compilation command 2644 can configure HCC 2660 to compile HIP source code 2630 using an HCC head and HIP / HCC runtime library. In response to HIP / HCC compilation command 2644, HCC 2660 can generate host executable code 2670(2) and HCC device executable code 2682. HCC device executable code 2682 can be a compiled version of device code contained in HIP source code 2630, which is executable on GPU 2692. GPU 2692 can be any processor optimized for parallel instruction processing, non-CUDA compatible, and HCC compatible. GPU 2692 may be developed by AMD Corporation of Santa Clara, CA. GPU 2692 may include a non-CUDA-enabled GPU 2692.

[0247] The following are for illustrative purposes only: Fig. Figure 26 shows three different flows that can be implemented in at least one embodiment to compile CUDA source code 2610 for execution on CPU 2690 and various devices. A direct CUDA flow can compile CUDA source code 2610 for execution on CPU 2690 and CUDA-enabled GPU 2694 without translating CUDA source code 2610 into HIP source code 2630. An indirect CUDA flow can translate CUDA source code 2610 into HIP source code 2630 and then compile the HIP source code 2630 for execution on CPU 2690 and CUDA-enabled GPU 2694. A CUDA / HCC flow can translate CUDA source code 2610 into HIP source code 2630 and then compile the HIP source code 2630 for execution on CPU 2690 and GPU 2692.

[0248] A direct CUDA flow that can be implemented is represented by dashed lines and a series of bubbles labeled A1-A3. As shown by bubble A1, CUDA compiler 2650 can receive CUDA source code 2610 and a CUDA compilation instruction 2648, which can configure CUDA compiler 2650 to compile CUDA source code 2610. CUDA source code 2610, which can be used in a direct CUDA flow, can be written in a CUDA programming language based on a programming language other than C++ (e.g., C, Fortran, Python, Java, etc.). In response to CUDA compilation instruction 2648, CUDA compiler 2650 can generate host executable code 2670(1) and CUDA device executable code 2684 (shown as bubble labeled A2). As shown as bubble labeled A3, host executable code 2670(1) and CUDA device executable code 2684 can be executed on CPU 2690 and CUDA-enabled GPU 2694, respectively.CUDA device executable code 2684 can contain binary code. CUDA device executable code 2684 can contain PTX code and can furthermore be compiled into binary code for a specific target device at runtime.

[0249] An indirect CUDA flow that can be implemented is represented by dashed lines and a series of bubbles labeled B1-B6. As shown by bubble B1, CUDA-to-HIP translation tool 2620 can receive CUDA source code 2610. As shown by bubble B2, CUDA-to-HIP translation tool 2620 can translate CUDA source code 2610 into HIP source code 2630. As shown by bubble B3, HIP compiler driver 2640 can receive HIP source code 2630 and determine that target device 2646 is CUDA-enabled.

[0250] As shown in bubble B4, HIP compiler driver 2640 can generate HIP / NVCC compilation instruction 2642 and can transfer both HIP / NVCC compilation instruction 2642 and HIP source code 2630 to CUDA compiler 2650. HIP / NVCC compilation instruction 2642 can configure CUDA compiler 2650 to compile HIP source code 2630 using a HIP-to-CUDA translation header and a CUDA runtime library. The HIP-to-CUDA translation header can translate any number of mechanisms (e.g., functions) specified in any number of HIP APIs into any number of mechanisms specified in any number of CUDA APIs. CUDA compiler 2650 can use HIP-to-CUDA translation header in conjunction with a CUDA runtime library that conforms to the CUDA runtime API 2602 to generate executable host code 2670(1) and executable CUDA device code 2684.In response to HIP / NVCC compilation instruction 2642, CUDA compiler 2650 can generate host executable code 2670(1) and CUDA device executable code 2684 (represented by bubble labeled B5). As shown by bubble labeled B6, host executable code 2670(1) and CUDA device executable code 2684 can be executed on CPU 2690 and CUDA-enabled GPU 2694, respectively. CUDA device executable code 2684 can contain binary code. CUDA device executable code 2684 can also contain PTX code and can furthermore be compiled at runtime into binary code for a specific target device.

[0251] A CUDA / HCC flow that can be implemented is represented by solid lines and a series of bubbles labeled C1-C6. As shown by bubble C1, CUDA-to-HIP translation tool 2620 can receive CUDA source code 2610. As shown by bubble C2, CUDA-to-HIP translation tool 2620 can translate CUDA source code 2610 into HIP source code 2630. As shown by bubble C3, HIP compiler driver 2640 can receive HIP source code 2630 and determine that target device 2646 is not CUDA-enabled.

[0252] HIP compiler driver 2640 can generate HIP / HCC compilation command 2644 and can transfer both HIP / HCC compilation command 2644 and HIP source code 2630 to HCC 2660 (represented by a bubble labeled C4). HIP / HCC compilation command 2644 can configure HCC 2660 to compile HIP source code 2630 using an HCC header and a HIP / HCC runtime library. The HIP / HCC runtime library can correspond to HIP runtime API 2632. The HCC header can include any number and type of interoperability mechanisms for HIP and HCC. In response to HIP / HCC compilation instruction 2644, HCC 2660 can generate host executable code 2670(2) and HCC device executable code 2682 (represented by bubble labeled C5). As represented by bubble labeled C6, host executable code 2670(2) and HCC device executable code 2682 can be executed on CPU 2690 and GPU 2692, respectively.

[0253] After CUDA source code 2610 has been translated into HIP source code 2630, HIP compiler driver 2640 can then be used to generate executable code for either CUDA-enabled GPU 2694 or GPU 2692 without re-executing CUDA-to-HIP translation tool 2620. CUDA-to-HIP translation tool 2620 can translate CUDA source code 2610 into HIP source code 2630, which is then stored in memory. HIP compiler driver 2640 can then configure HCC 2660 to generate host executable code 2670(2) and HCC device executable code 2682 based on HIP source code 2630. In at least one embodiment, HIP compiler driver 2640 subsequently configures CUDA compiler 2650 to generate host executable code 2670(1) and CUDA device executable code 2684 based on stored HIP source code 2630.

[0254] An example kernel can be generated by the CUDA-to-HIP translation tool 2620 from Fig. 26 according to at least one embodiment. CUDA source code 2610 partitions an overall problem that a given kernel is to solve into relatively coarse subproblems that can be solved independently using thread blocks. Each thread block contains an arbitrary number of threads. Each subproblem can be partitioned into relatively fine pieces that can be solved cooperatively in parallel by threads within a thread block. Threads within a thread block can cooperate by sharing data through shared memory and by synchronizing execution to coordinate memory accesses.

[0255] CUDA source code 2610 can organize thread blocks associated with a given kernel into a one-dimensional, two-dimensional, or three-dimensional lattice of thread blocks. Each thread block can contain any number of threads, and a lattice can contain any number of thread blocks.

[0256] A kernel can be a function in device code defined using a "__global_" declaration specifier. The dimension of a grid that executes a kernel for a given kernel call and associated streams can be specified using a CUDA kernel start syntax. The CUDA kernel start syntax is defined as "KernelName<<<GridSize, BlockSize, SharedMemorySize, Stream> >(KernelArguments);” specifies. An execution configuration syntax can include a construct “<<<...>>” inserted between a kernel name (“KernelName”) and a list of kernel arguments (“KernelArguments”) enclosed in parentheses. CUDA kernel startup syntax can include a CUDA startup function syntax instead of an execution configuration syntax.

[0257] `GridSize` can be of type `dim3` and specify the dimension and size of a grid. Type `dim3` can be a CUDA-defined structure containing unsigned integers `x`, `y`, and `z`. If `z` is not specified, it defaults to one. If `y` is not specified, it defaults to one. The number of thread blocks in a grid can be equal to the product of `GridSize.x`, `GridSize.y`, and `GridSize.z`. `BlockSize` can be of type `dim3` and specify the dimension and size of each thread block. The number of threads per thread block can be equal to the product of `BlockSize.x`, `BlockSize.y`, and `BlockSize.z`. Each thread executing a kernel can be given a unique thread ID, accessible within the kernel through a built-in variable (e.g., `threadIdx`).

[0258] Regarding CUDA kernel startup syntax, `SharedMemorySize` can be an optional argument that specifies the number of bytes in shared memory that are dynamically allocated per thread block for a given kernel call, in addition to statically allocated memory. By default, `SharedMemorySize` can be zero. Similarly, `Stream` can be an optional argument that specifies an associated stream and is zero by default to specify a standard stream. A stream can be a sequence of instructions (possibly issued by different host threads) that are executed sequentially. Different streams can execute instructions out of order, either sequentially or concurrently.

[0259] CUDA source code 2610 can contain a kernel definition for an example kernel "MatAdd" and a main function. The main function can be host code that runs on a host and includes a kernel call that causes the kernel MatAdd to run on a device. Kernel MatAdd can add two matrices A and B of size NxN, where N is a positive integer, and store the result in a matrix C. The main function can define a ThreadsPerBlock variable as 16 by 16 and a numBlocks variable as N / 16 by N / 16. The main function can then make the kernel call "MatAdd<<numBlocks, threadsPerBlock> >(A, B, C);“ specify. According to CUDA kernel startup syntax, the kernel MatAdd can be executed using a grid of thread blocks with a dimension of N / 16 by N / 16, where each thread block has a dimension of 16 by 16.Each thread block can contain 256 threads, a grid can be created with enough blocks to have one thread per matrix element, and each thread in such a grid can execute the kernel MatAdd to perform pairwise addition.

[0260] While CUDA source code 2610 is translated into HIP source code 2630, the CUDA-to-HIP translation tool 2620 can translate any kernel call in CUDA source code 2610 from CUDA kernel launch syntax to HIP kernel launch syntax and can convert any number of other CUDA calls in source code 2610 into any number of other functionally similar HIP calls. HIP kernel launch syntax can be specified as "hipLaunchKernelGgle(KernelName, GridSize, BlockSize, SharedMemorySize, Stream, KernelArguments);". Each of the following elements—KernelName, GridSize, BlockSize, ShareMemorySize, Stream, and KernelArguments—can have the same meaning in the HIP kernel startup syntax as in the CUDA kernel startup syntax (described earlier herein). The SharedMemorySize and Stream arguments may be required in the HIP kernel startup syntax and may be optional in the CUDA kernel startup syntax.

[0261] A section of the HIP source code 2630 can be identical to a section of the depicted CUDA source code 2610, except for a kernel call that causes the kernel MatAdd to run on a device. Kernel MatAdd can be defined in the HIP source code 2630 with the same "__global_" declaration specifier used to define Kernel MatAdd in the CUDA source code 2610. A kernel call in the HIP source code 2630 can be "hipLaunchKernelGgle(MatAdd, numBlocks, threadsPerBlock, 0, 0, A, B, C);", while a corresponding kernel call in the CUDA source code 2610 might be "MatAdd<<numBlocks, threadsPerBlock> >(A, B, C);“ is.

[0262] Other implementations are being considered and can be carried out similarly to the CUDA and HIP implementations above, such as oneAPI, OpenCL, and other programming platforms. Code can be translated in either direction. For example, CUDA can be translated to HIP, and CUDA can be translated to OpenCL. SnuCL-Tr and CUCL can be used to translate OpenCL to CUDA and CUDA to OpenCL, respectively. Compiled code or intermediate representations (e.g., CUDA-PTX code) can also be translated to run on other processor platforms (e.g., AMD or Intel). For example, PTX code can be translated to run on Intel or AMD processors using a translation tool such as ZLUDA.

[0263] One or more of the techniques described herein can utilize a oneAPI programming model. A oneAPI programming model can refer to a programming model for interacting with various compute accelerator architectures. OneAPI can refer to an application programming interface (API) designed to interact with various compute accelerator architectures. A oneAPI programming model can utilize a DPC++ programming language. A DPC++ programming language can refer to a high-level language for data-parallel programming productivity. A DPC++ programming language can be based, at least partially, on C and / or C++ programming languages. A oneAPI programming model can be a programming model such as, but not limited to, those developed by Intel Corporation of Santa Clara, CA.

[0264] OneAPI and / or the oneAPI programming model can be used to interact with various accelerator, GPU, processor, and / or variations thereof architectures. OneAPI can include a set of libraries that implement various functionalities. OneAPI can include at least one oneAPI DPC++ library, oneAPI math core library, oneAPI data analysis library, oneAPI deep neural network library, oneAPI collection communications library, oneAPI threading component library, oneAPI video processing library, and / or variations thereof.

[0265] A oneAPI-DPC++ library, also known as oneDPL, can be a library that implements algorithms and functions to accelerate DPC++ core programming. OneDPL can implement one or more Standard Template Library (STL) functions. OneDPL can implement one or more parallel STL functions. OneDPL can provide a set of library classes and functions, such as, but not limited to, parallel algorithms, iterators, function object classes, range-based APIs, and / or variations thereof. OneDPL can implement one or more classes and / or functions from a C++ standard library. OneDPL can implement one or more random number generator functions.

[0266] A oneAPI mathematics core library, also known as oneMKL, can be a library that implements various optimized and parallelized routines for different mathematical functions and / or operations. OneMKL can implement one or more Basic Linear Algebra Subprograms (BLAS) and / or Linear Algebra Package (LAPACK) dense linear algebra routines. OneMKL can implement one or more sparse BLAS linear algebra routines. OneMKL can implement one or more random number generators (RNGs). OneMKL can implement one or more vector mathematics (VM) routines for mathematical operations on vectors. OneMKL can implement one or more Fast Fourier Transform (FFT) functions.

[0267] A oneAPI data analysis library, also known as oneDAL, can comprise a library that implements various data analysis applications and distributed computations. OneDAL can implement various algorithms for preprocessing, transformation, analysis, modeling, validation, and decision-making for data analysis, in batch, online, and distributed processing modes. OneDAL can implement various C++ and / or Java APIs and various connectors to one or more data sources. OneDAL can implement DPC++ API extensions to a standard C++ interface and enables GPU utilization for various algorithms.

[0268] A oneAPI Deep Neural Network library, also known as oneDNN, can include a library that implements various deep learning functions. OneDNN can implement various neural network, machine learning, and deep learning functions, algorithms, and / or variations thereof.

[0269] A oneAPI collective communication library, also known as oneCCL, can include a library that implements various applications for deep learning and machine learning workloads. OneCCL can be built on top of lower-level communication middleware, such as, but not limited to, Message Passing Interface (MPI) and libfabrics. OneCCL can enable a set of deep learning-specific optimizations, such as, but not limited to, prioritization, persistent operations, out-of-order execution, and / or variations thereof. OneCCL can implement various CPU and GPU functions.

[0270] A oneAPI threading component library, also known as oneTBB, can include a library that implements various parallelized processes for different applications. OneTBB can be used for task-based, shared parallel programming on a single host. OneTBB can implement generic parallel algorithms. OneTBB can implement concurrent containers. OneTBB can implement a scalable memory allotment. OneTBB can implement a job-stealing task scheduler. OneTBB can implement low-level synchronization primitives. OneTBB can be compiler-independent and usable on various processors, such as, but not limited to, GPUs, PPUs, CPUs, and / or variations thereof.

[0271] A oneAPI video processing library, also known as oneVPL, can include a library used to accelerate video processing in one or more applications. OneVPL can implement various video decoding, encoding, and processing functions. OneVPL can implement various functions for media pipelines on CPUs, GPUs, and other accelerators. OneVPL can implement device discovery and selection in media-centric and video analytics workloads. OneVPL can implement API primitives for zero-copy buffer utilization.

[0272] A oneAPI programming model can utilize a DPC++ programming language. A DPC++ programming language can include functionally similar versions of CUDA mechanisms for defining device code and distinguishing between device code and host code. A DPC++ programming language can include a subset of the functionality of a CUDA programming language. One or more CUDA programming model operations can be performed using a oneAPI programming model and a DPC++ programming language.

[0273] Each application programming interface (API) described herein can be compiled into one or more instructions, operations, or any other signal by a compiler, interpreter, or other software tool. Compilation can involve generating one or more machine-executable instructions, operations, or other signals from source code. An API compiled into one or more instructions, operations, or other signals, when executed, can cause one or more processors, such as, but not limited to, processors used in, for example, Fig. 11- Fig. 23, or any other logic circuit further described herein, to perform one or more arithmetic operations.

[0274] In at least one embodiment, translation tools described elsewhere herein, such as, but not limited to, one or more circuits to translate CUDA code, to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens used by one or more application programming interfaces, to execute one or more neural networks generating tokens mapped using the tables on HIP, oneAPI, OpenCL, or any other language used to perform any of the operations described above or elsewhere herein.One or more circuits can be configured by software to translate CUDA code to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces, or to run one or more neural networks that generate tokens mapped using the tables to HIP, oneAPI, OpenCL, or any other language used to perform any of the operations described above or elsewhere herein. AUTONOMOUS VEHICLE

[0275] Fig. Figure 27 illustrates an example of an autonomous vehicle 2700 according to at least one embodiment. The autonomous vehicle 2700 (hereinafter referred to alternatively as "vehicle 2700") can be a passenger vehicle, such as, but not limited to, a car, a truck, a bus, and / or another type of vehicle that carries one or more passengers. In at least one embodiment, the vehicle 2700 can be a semi-trailer truck used for transporting cargo. Vehicle 2700 can be an aircraft, a robotic vehicle, or another type of vehicle.

[0276] Autonomous vehicles can be described in terms of automation levels defined by the National Highway Traffic Safety Administration (“NHTSA”), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (“SAE”) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and previous and future versions of this standard). In at least one embodiment, Vehicle 2700 can be capable of functionality corresponding to one or more of the Level 1 through Level 5 autonomous driving levels. For example, in at least one embodiment, Vehicle 2700 can be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0277] Vehicle 2700 can include components such as, but not limited to, a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 2700 can include a propulsion system 2750, such as an internal combustion engine, a hybrid electric power plant, a purely electric motor, and / or another type of propulsion system. The propulsion system 2750 can be connected to a drivetrain of Vehicle 2700, which may include a transmission to enable propulsion of Vehicle 2700. The propulsion system 2750 can be controlled in response to receiving signals from a throttle / accelerator(s) 2752.

[0278] A steering system 2754, which may include a steering wheel, is used to steer vehicle 2700 (e.g., along a desired path or route) when propulsion system 2750 is in operation (e.g., when vehicle 2700 is in motion). Steering system 2754 can receive signals from steering actuator(s) 2756. A steering wheel may be optional for full automation (Level 5) functionality. A brake sensor system 2746 can be used to operate vehicle brakes in response to receiving signals from brake actuator(s) 2748 and / or brake sensors.

[0279] Controller(s) 2736, which may include one or more system-on-chips (“SoCs”) and / or graphics processing units (“GPU(s)”), can provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 2700. For example, controller(s) 2736 can send signals to operate vehicle brakes via brake actuator(s) 2748, to operate steering system 2754 via steering actuator(s) 2756, and to operate propulsion system 2750 via throttle / accelerator(s) 2752. Controller(s) 2736 can include one or more onboard (e.g., integrated) computing devices that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in operating vehicle 2700.Control(s) 2736 may include a first control for autonomous driving functions, a second control for functional safety functions, a third control for artificial intelligence functionality (e.g., computer vision), a fourth control for infotainment functionality, a fifth control for emergency redundancy, and / or other controls. A single control may handle two or more of the above functionalities; two or more controls may handle a single functionality and / or any combination thereof.

[0280] Controller(s) 2736 can provide signals to control one or more components and / or systems of vehicle 2700 in response to sensor data received from one or more sensors (e.g. sensor inputs). Sensor data can be obtained, for example, from sensor(s) 2758 (e.g., global positioning system sensor(s)), radar sensor(s) 2760, ultrasonic sensor(s) 2762, lidar sensor(s) 2764, inertial measurement unit (“IMU”) sensor(s) 2766 (e.g., accelerometer, gyroscope, magnetic compass(s), magnetometer, etc.), microphone(s) 2796, stereo camera(s) 2768, long-range camera(s) 2770 (e.g., fisheye cameras), infrared camera(s) 2772, surround camera(s) 2774 (e.g., 360-degree cameras), long-range camera(s) 2798, medium-range camera(s) 2776, speed sensor(s) 2744 (e.g., for measuring the vehicle's speed). 2700), vibration sensor(s) 2742, steering sensor(s) 2740, brake sensor(s) (e.g.as part of the brake sensor system 2746) and / or other sensor types.

[0281] One or more of the controller(s) 2736 can receive inputs (e.g., represented by input data) from an instrument cluster 2732 of vehicle 2700 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 2734, an audible announcer, a loudspeaker, and / or other components of vehicle 2700. Outputs can include information such as, but not limited to, vehicle speed, velocity, time, map data (e.g., a high-resolution map (not shown)), location data (e.g., the location of vehicle 2700, such as, but not limited to, on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by controller(s) 2736, etc. For example, the HMI display 2734 can show information about the presence of one or more objects (e.g.,a road sign, warning sign, traffic light change, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, entering exit 34B in two miles, etc.).

[0282] Each of the components, features and systems of vehicle 2700 in Fig. Vehicle 2700 can be connected via a bus 2702. Bus 2702 can include a CAN data interface (referred to herein alternatively as a "CAN bus"). A CAN bus can be a network within Vehicle 2700 that is used to assist in controlling various features and functionality of Vehicle 2700, such as, but not limited to, actuation of brakes, acceleration, braking, steering, windshield wipers, etc. Bus 2702 can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). Bus 2702 can be read to obtain steering wheel angle, ground speed, engine revolutions per minute ("RPMs"), button positions, and / or other vehicle status indicators. Bus 2702 can be a CAN bus that is ASIL B compliant.

[0283] In addition to or as an alternative to CAN, FlexRay and / or Ethernet protocols can be used. There can be any number of buses that make up Bus 2702, which can include zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. Two or more buses can be used to perform different functions and / or for redundancy. For example, a first bus can be used for collision avoidance functionality, and a second bus can be used for actuation control. Each bus of Bus 2702 can communicate with any of the components of Vehicle 2700, and two or more buses of Bus 2702 can communicate with corresponding components.Each of any number of system(s) on chip(s) (“SoC(s)”) 2704 (such as, but not limited to, SoC 2704(A) and SoC 2704(B)), each of the controller(s) 2736 and / or each computer within vehicle can have access to the same input data (e.g., inputs from sensors of vehicle 2700) and can be connected to a common bus, such as CAN bus.

[0284] Any number of cameras can be placed at any choice of camera locations and fields of view for the autonomous vehicle 2500 of Fig. 25A shall be positioned according to at least one embodiment. Cameras and their respective fields of view may be exemplary embodiments and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or cameras may be located at different locations on the vehicle 2500.

[0285] Camera types may include digital cameras designed for use with components and / or systems of the Vehicle 2500. Camera(s) may operate at Automotive Safety Integrity Level (“ASIL”) B and / or another ASIL. Camera types may be capable of any frame rate, such as, but not limited to, 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. Cameras may be capable of using roller shutters, global shutters, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a Red Clear Clear Clear (“RCCC”) color filter array, a Red Clear Clear Blue (“RCCB”) color filter array, a Red Blue Green Clear (“RBGC”) color filter array, a Foveon X3 color filter array, a Bayer Sensors (“RGGB”) color filter array, a monochrome sensor color filter array, and / or another type of color filter array.Clear pixel cameras, such as, but not limited to, cameras with an RCCC, an RCCB and / or an RBGC color filter array, can be used in an effort to increase light sensitivity.

[0286] One or more cameras can be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign recognition, and intelligent headlight control. One or more cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0287] One or more cameras can be mounted in an assembly, such as, but not limited to, a custom-designed (three-dimensional ("3D") printed) assembly to filter out stray light and reflections from within the vehicle (e.g., reflections from the dashboard, reflected in the windshield wipers) that can interfere with camera image acquisition. With regard to vane mirror mounting assemblies, vane mirror assemblies can be custom 3D printed so that a camera mounting plate conforms to the shape of a vane mirror. Camera(s) can be integrated into vane mirrors. For side-view cameras, camera(s) can also be integrated into four pillars at each corner of a cabin.

[0288] Cameras with a field of view that includes parts of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to help identify forward paths and obstacles, and to help provide, with the aid of one or more of the controller(s) and / or control SoCs, information critical for generating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many similar ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions, such as, but not limited to, traffic sign recognition.

[0289] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform incorporating a CMOS (Complementary Metal Oxide Semiconductor) color imager. A long-range camera 2770 can be used to detect objects approaching from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Any number (including zero) of long-range cameras 2770 can be mounted on the vehicle 2700. Any number of long-range camera(s) 2798 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. Long-range camera(s) 2798 can also be used for object detection and classification, as well as basic object tracking.

[0290] Any number of stereo camera(s) 2768 can also be included in a forward-facing configuration. One or more of the stereo camera(s) 2768 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle 2700's environment, including a distance estimate for all points in an image. One or more of the stereo camera(s) 2768 can include a compact stereo vision sensor, which can include two camera lenses (one each on the left and right) and an image processing chip that measures the distance from the vehicle 2700 to the target object and generates information (e.g.,Metadata) can be used to activate autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s) 2768 can be used in addition to or as an alternative to those described here.

[0291] Cameras with a field of view that includes parts of the surroundings to the sides of the vehicle 2700 (e.g., side-view cameras) can be used for surround view, providing information that is used to create and update an occupancy grid and to generate side-impact collision warnings. For example, surround-view camera(s) 2774 (e.g., four surround-view cameras) could be positioned on the vehicle 2700. The surround-view camera(s) 2774 can include any number and combination of wide-view cameras, fisheye camera(s), 360-degree camera(s), and / or similar cameras. For example, four fisheye cameras could be positioned on the front, rear, and sides of the vehicle 2700. The vehicle 2700 can use three surround-view camera(s) 2774 (e.g. left, right and rear) and can use one or more other camera(s) (e.g. a forward-facing camera) as a fourth surround-view camera.

[0292] Cameras with a field of view that includes parts of the area behind the vehicle 2700 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear collision warnings, and creating and updating an occupancy grid. A wide variety of cameras can be used, including, but not limited to, cameras that can also be used as forward-facing cameras (e.g., long-range cameras 2798 and / or medium-range cameras 2776, stereo cameras 2768, infrared cameras 2772, etc.), as described herein.

[0293] Vehicle 2700 can include any number of SoCs 2704 or other processors described elsewhere herein, such as, but not limited to, processors and / or components designed for Fig. 11- Fig. Figure 23 illustrates and describes the following. Each of the SoCs 2704 can include central processing units (“CPU(s)”) 2706, graphics processing units (“GPU(s)”) 2708, processor(s) 2710, cache(s) 2712, accelerator(s) 2714, data storage(s) 2716, and / or other components and features not illustrated. SoC(s) 2704 can be used to control vehicle 2700 in a variety of platforms and systems. For example, SoC(s) 2704 can be combined in a system (e.g., the vehicle 2700 system) with a high-resolution (“HD”) map 2722, which can receive map updates and / or updates via network interface 2724 from one or more servers (not shown).

[0294] CPU(s) 2706 can include a CPU cluster or CPU complex (referred to herein alternatively as a "CCPLEX"). CPU(s) 2706 can include multiple cores and / or Level 2 (“L2”) caches. For example, CPU(s) 2706 can include eight cores in a coherent multiprocessor configuration. CPU(s) 2706 can include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2-megabyte (MB) L2 cache). CPU(s) 2706 (e.g., CCPLEX) can be configured to support concurrent cluster operations, allowing any combination of CPU(s) 2706 clusters to be active at any given time.

[0295] One or more of the CPU(s) 2706 can implement power management capabilities that include one or more of the following features: Individual hardware blocks can be automatically clocked when idle to save dynamic power; each core can be clocked when such core is not actively executing due to the execution of Wait for Interrupt ("WFI") / Wait for Event ("WFE") instructions; each core can be clocked independently; each core cluster can be clocked independently if all cores are clockable; and / or each core cluster can be clocked independently if all cores are clockable.CPUs 2706 can further implement an extended power state management algorithm, where allowed power states and expected wake-up times can be specified, and hardware / microcode determines which best power state to input for the core, cluster, and CCPLEX. Processing cores can support simplified power state entry sequences in software, offloading work to microcode.

[0296] GPU(s) 2708 may include an integrated GPU (referred to herein alternatively as an "iGPU"). GPU(s) 2708 may be programmable and efficient for parallel workloads. GPU(s) 2708 may use an extended Tensor instruction set. GPU(s) 2708 may include one or more streaming microprocessors, each of which may include a level one ("L1") cache (e.g., an L1 cache of at least 96 KB), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache of 512 KB). GPU(s) 2708 may include at least eight streaming microprocessors. GPU(s) 2708 may use compute application programming interface(s) (API(s)). GPU(s) 2708 can use one or more parallel computing platforms and / or programming models (e.g. NVIDIA's CUDA model).Streaming microprocessors can be referred to as streaming multiprocessors (“SMs”), stream processors (“SPs”), stream processing units (“SPUs”), compute units (“CUs”), execution units (“EUs”) and / or slices, where a slice in this context can refer to a portion of processing resources in a processing unit (e.g., 16 cores, a ray tracing unit, a thread director, or a scheduler).

[0297] One or more of the GPU(s) 2708 can be power-optimized for best performance in automotive and embedded applications. For example, GPU(s) 2708 could be manufactured using FinFET (Fin-Fin) circuitry. Each streaming microprocessor can include a number of mixed-precision processing cores partitioned into multiple blocks. For example, 64 FP32 cores and 32 FP64 cores could be partitioned into four processing blocks. Each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA mixed-precision Tensor Cores for deep learning matrix arithmetic, a level zero (L0) instruction cache, a scheduler (such as a warp scheduler) or sequencer, a dispatch unit, and / or a 64 KB register file.Streaming microprocessors can include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mix of computational and addressing operations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combined L1 data cache and shared memory to improve performance while simplifying programming.

[0298] One or more of the GPU(s) 2708 can include high-bandwidth memory (“HBM”) and / or a 16 GB HBM2 memory subsystem to provide peak memory bandwidth of approximately 900 GB / second in some examples. In addition to or as an alternative to HBM memory, synchronous graphics random-access memory (“SGRAM”) can be used, such as, among others, synchronous graphics double-rate type five random-access memory (“GDDR5”).

[0299] GPU(s) 2708 can incorporate unified memory technology. Support for Address Translation Services (“ATS”) can be used to allow GPU(s) 2708 to directly access CPU(s) 2706 page tables. If a GPU(s) 2708 memory management unit (MMU) experiences an error, an address translation request can be sent to CPU(s) 2706. In response, two CPU(s) 2706 can search their page tables for a virtual-to-physical mapping for an address and send the translation back to GPU(s) 2708. Unified memory technology can enable a single unified virtual address space for memory from both CPU(s) 2706 and GPU(s) 2708, simplifying GPU(s) 2708 programming and porting applications to GPU(s) 2708.

[0300] GPU(s) 2708 can include any number of access counters that track the frequency of GPU(s) 2708 accesses to the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses pages most frequently, thereby improving efficiency for memory areas shared between processors.

[0301] One or more SoC(s) 2704 can include any number of Cache(s) 2712, including those described herein. For example, Cache(s) 2712 could include a Level 3 (“L3”) cache available to both CPU(s) 2706 and GPU(s) 2708 (e.g., one connected to CPU(s) 2706 and GPU(s) 2708). Cache(s) 2712 can include, but are not limited to, a write-back cache capable of tracking row states, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). An L3 cache can include 4 MB of memory or more, depending on the implementation, although smaller cache sizes are possible.

[0302] One or more SoC(s) 2704 can include one or more Accelerators 2714 (e.g., hardware accelerators, software accelerators, or a combination thereof). SoC(s) 2704 can include a hardware acceleration cluster, which may contain optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4 MB SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other computations. A hardware acceleration cluster can be used to complement GPU(s) 2708 and offload some tasks from GPU(s) 2708 (e.g., to free up more GPU cycles for other tasks). Accelerator(s) 2714 could be used for targeted workloads (e.g., perception, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.) that may be stable enough to be amenable to acceleration.A CNN can include a region-based or regional convolutional neural network (“RCNNs”) and fast RCNNs (e.g., as used for object detection) or another type of CNN.

[0303] Accelerator(s) 2714 (e.g., hardware acceleration clusters) can include one or more deep learning accelerators (“DLAs”). DLAs can include one or more tensor processing units (“TPUs”), which can be configured to provide an additional ten trillion operations per second for deep learning applications and inference, such as TPUs in Fig. 21. TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). DLA(s) can further be optimized for a specific set of neural network types and floating-point operations, as well as inference. Designed DLA(s) can deliver more performance per millimeter than a typical general-purpose GPU and typically far exceeds the performance of a CPU. TPU(s) can perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.DLA(s) can quickly and efficiently execute neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, including, for example: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security-related events.

[0304] DLA(s) can perform any function of GPU(s) 2708, and by using an inference accelerator, for example, a designer can target either DLA(s) or GPU(s) 2708 for any given function. For example, a designer can focus CNN processing and floating-point operations on DLA(s) and leave other functions to GPU(s) 2708 and / or accelerator(s) 2714.

[0305] Accelerator(s) 2714 may include a programmable vision accelerator (“PVA”), which may alternatively be referred to herein as a computer vision accelerator. PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (“ADAS”) 2738, autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. PVA may provide a balance between performance and flexibility. For example, each PVA may include any number of reduced instruction set computer (“RISC”) cores, direct memory access (“DMA”) cores, and / or any number of vector processors.

[0306] RISC cores can interact with image sensors (e.g., image sensors from any camera described herein), image signal processor(s), and so on. Each RISC core can include any amount of memory. RISC cores can use any of a number of protocols, depending on the implementation. RISC cores can run a real-time operating system (RTOS). RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, RISC cores could include an instruction cache and / or tightly coupled RAM.

[0307] DMA can enable PVA components to access system memory independently of the CPU(s) 2706. DMA can support any number of features used to provide optimization for a PVA, including support for multidimensional addressing and / or circular addressing. DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0308] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. A PVA can include a PVA core and two vector processing subsystem partitions. A PVA core can include a processor subsystem, DMA machine(s) (e.g., two DMA machines), and / or other peripherals. A vector processing...

Claims

[1] A processor comprising: one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces (APIs). [2] The processor according to claim 1, wherein a single entry displays a plurality of entries of a single table of one or more tables, given a single current state of a state representation of accepted input tokens generated by the one or more neural networks for a single API of one or more APIs, whether generation of a single token by the one or more neural networks results in input that is not acceptable for the single API. [3] The processor according to claim 2, wherein the one or more circuits cause respective portions of the individual table to be created in parallel by assigning respective threads to a plurality of threads of a graphics processing unit (GPU) to generate entries of the respective portions for respective subsets of a vocabulary of tokens of the one or more neural networks. [4] The processor according to claim 1, wherein during inference operations of the one or more neural networks the one or more circuits further cause an output probability assigned to a single token within a neural network of the one or more neural networks to be set to zero, based at least partially on examining (a) a single table of the one or more tables and (b) a current state of a state representation of accepted input tokens of a single API of the one or more APIs. [5] The processor according to claim 1, wherein the one or more circuits cause at least one table of the one or more tables to be generated at least partially based on analysis of a first vector and a second vector, wherein the first vector comprises a list of bytes that are generated by converting tokens of a token vocabulary of the one or more neural networks into corresponding bytes, and wherein the second vector comprises start indices within the first vector of individual tokens of the token vocabulary. [6] The processor according to claim 5, wherein the analysis of the first vector and the second vector is performed in parallel by assigning respective threads to a plurality of threads of a graphics processing unit (GPU) to analyze respective subsets of the first vector and the second vector. [7] The processor according to claim 1, wherein the one or more circuits cause at least one table of the one or more tables to be at least partially generated by mapping to respective transition identifiers of a state transition representation of accepted input tokens of the one or more APIs, of identifiers of byte representations of the accepted input tokens. [8] The processor according to claim 7, wherein the mapping to the respective transition identifiers is performed in parallel by assigning respective threads to a plurality of threads of a graphics processing unit (GPU) to map respective subsets of the identifiers of the byte representations of the accepted input tokens. [9] A process comprising: Causing one or more tables to map one or more tokens, generated by one or more neural networks, into one or more tokens that can be used by one or more application programming interfaces (APIs). [10] The method according to claim 9, wherein a single entry displays a plurality of entries of a single table of one or more tables, given a single current state of a state representation of accepted input tokens generated by the one or more neural networks for a single API of one or more APIs, whether generation of a single token by the one or more neural networks results in input that is acceptable for the single API. [11] The method according to claim 9, further comprising: During inference operations of one or more neural networks, cause an output probability assigned to a single token within one or more neural networks to be set to zero, based at least partially on examining (a) a single table of one or more tables and (b) a current state of a state representation of accepted input tokens of a single API of one or more APIs. [12] The method according to claim 9, further comprising: Causing a text representation of accepted input tokens of a single API of one or more APIs to be converted into a state representation of accepted input tokens, using the state representation to generate at least one table of one or more tables. [13] The method according to claim 12, wherein the text representation comprises one or more of: (a) a regular expression or (b) a JavaScript Object Notation (JSON) descriptor of the accepted inputs. [14] The method according to claim 9, wherein a single neural network of one or more neural networks is incorporated into a Large Language Model (LLM) agent, and wherein the LLM agent is configured for use to prepare a response to an input prompt of a tool that implements a single API of one or more APIs. [15] The method according to claim 9, further comprising: Causing a single table of one or more tables to be generated, based at least partially on determining that a single API should be added to a set of APIs that are called via an output generated by the one or more neural networks. [16] A system comprising: one or more processors, comprising one or more circuits to cause one or more tables to map one or more tokens generated by one or more neural networks into one or more tokens to be used by one or more application programming interfaces (APIs); and one or more memories for storing parameters associated with the one or more neural networks. [17] The system according to claim 16, wherein a single entry displays a plurality of entries of a single table of one or more tables, given a single current state of a state representation of accepted input tokens generated by the one or more neural networks for a single API of one or more APIs, whether the generation of a single token by the one or more neural networks will result in input that is not acceptable for the single API. [18] The system according to claim 17, wherein the one or more circuits cause respective parts of the individual table to be created in parallel by assigning respective threads to a plurality of threads to generate entries of the respective parts for respective subsets of a vocabulary of tokens of the one or more neural networks. [19] The system according to claim 16, wherein during inference operations of the one or more neural networks the one or more circuits further cause an output probability assigned to a single token within a neural network of the one or more neural networks to be set to zero, based at least partially on examining (a) a single table of the one or more tables and (b) a current state of a state representation of accepted input tokens of a single API of the one or more APIs. [20] The system according to claim 16, wherein the one or more circuits cause at least one table of the one or more tables to be generated at least partially based on analysis of a first vector and a second vector, wherein the first vector comprises a list of bytes that are generated by converting tokens of a token vocabulary of the one or more neural networks into corresponding bytes, and wherein the second vector comprises start indices within the first vector of individual tokens of the token vocabulary.

Images