Collective perception of ai model traffic

Abstract

Systems and techniques for set-aware processing of AI model traffic are disclosed. The techniques include generating a first network packet having first metadata. The first metadata indicates a first association with a plurality of parallel processing units for performing a distributed artificial intelligence (AI) operation. The techniques also include, after generating the first network packet, generating a second network packet having second metadata. The second metadata indicates a second association with the plurality of parallel processing units for performing the distributed AI operation. The techniques also include determining, based on a comparison of the second metadata to the first metadata, that the second network packet has a higher priority than the first network packet. The techniques also include providing the second network packet to a networked device prior to providing the first network packet.

Description

Technical Field

[0001] At least one embodiment relates to distributed computing operations, and more specifically, to network communication between distributed processors. Background Technology

[0002] Massively parallel artificial intelligence (AI) models break down computation into smaller, not necessarily uniform, parts that are executed in parallel. In some cases, multiple processors can be used to perform the computation. In other cases, the processors are hosted by multiple systems. These multiple processors can be connected to each other via one or more networks and can communicate with each other via those networks.

[0003] A process running on a first processor may depend on processes running on one or more other processors. Processors can use network packets to transfer interdependent data and / or results over a network. The coordinated communication and operation between processors is called a "collective". Attached Figure Description

[0004] Figure 1 This is a block diagram of an exemplary distributed artificial intelligence (AI) network for collective-aware processing of AI model traffic according to at least one embodiment.

[0005] Figure 2 This is a block diagram of an exemplary artificial intelligence (AI) system for aggregate-aware processing of AI model traffic according to at least one embodiment.

[0006] Figure 3 This is a block diagram of an exemplary artificial intelligence (AI) metadata-aware switch for aggregate-aware processing of AI model traffic according to at least one embodiment.

[0007] Figure 4A This is an example timing diagram of an artificial intelligence (AI) system providing rules to an AI metadata-aware switch according to at least one embodiment.

[0008] Figure 4B This is an example timing diagram of an artificial intelligence (AI) system responding to a query from an AI metadata-aware switch according to at least one embodiment.

[0009] Figure 5 This is a flowchart of an example method for ensemble-aware processing of AI model traffic according to at least one embodiment.

[0010] Figure 6 This is a flowchart of an example method for ensemble-aware processing of AI model traffic according to at least one embodiment.

[0011] Figure 7 The illustration shows a computer system according to at least one embodiment.

[0012] Figure 8 The illustration shows a computer system according to at least one embodiment.

[0013] Figure 9A This is a block diagram of an example generative language model system applicable to implementing some embodiments of the present disclosure.

[0014] Figure 9B This is a block diagram of an example generative language model that includes a transformer encoder-decoder suitable for implementing some embodiments of this disclosure.

[0015] Figure 9C This is a block diagram of an example generative language model that includes a pure decoder transformer architecture suitable for implementing some embodiments of this disclosure. Detailed Implementation

[0016] Massively parallel artificial intelligence (AI) models break down computation into smaller, not necessarily uniform, parts that are executed in parallel. In some cases, multiple processors can be used to perform the computation, which can be hosted by multiple systems and interconnected via one or more networks.

[0017] A process running on a first processor may depend on processes running on one or more other processors. Processors can use network packets to transmit interdependent data and / or results over a network. The coordinated communication and operation between processors is called a "set".

[0018] In many cases, network packets transmitted between processors are treated equally. This can lead to a situation where time-critical communication between processors is slowed down by non-critical communication. For example, a particular process may need to wait and remain idle until it receives the data it depends on (e.g., via the network), resulting in a waste of time and computing resources.

[0019] Various aspects of this disclosure address the aforementioned and other shortcomings by providing ensemble-aware processing of AI model traffic. For example, developers of parallel processing unit kernels who write code for artificial intelligence (AI) kernels can identify specific regions of their code as "high priority." When the code is compiled and executed using a parallel processing unit, network packets created from these high-priority regions of the code for transmission to other processors can be marked as high priority. In some embodiments, one or more headers of the network packets may be modified, or headers may be added, to mark packets as high priority.

[0020] In some embodiments, network packets generated by different regions of the code can be tagged to identify one or more attributes of the data carried by the network packets. This tagging may include one or more labels, flags, or identifiers included in the metadata of the network packets. The tagging may indicate attributes of the network packets and / or their data, such as priority, type of set operation, data type, etc. For example, a network packet generated by a first code region may be tagged with first metadata indicating the type of data transmitted by that first code region, and a network packet generated by a second code region may be tagged with second metadata indicating the type of data transmitted by that second code region.

[0021] When network packets are being processed by network devices (e.g., network interface controllers (NICs), routers, switches, etc.), packets identified as high-priority are processed first. For example, high-priority network packets may be processed before other network packets in the network queue. High-priority network packets may also be routed based on their priority. In some cases, if a particular router or switch in the network becomes congested, non-high-priority packets may be dropped or delayed to reduce congestion and allow high-priority packets to be transmitted. In other cases, if a particular router or switch in the network becomes congested, packets marked as belonging to a low-volume set may still be buffered instead of dropped to avoid retransmission.

[0022] In some embodiments, when kernel code is compiled and executed using a parallel processing unit, it may send or cause to be sent (e.g., by a process running on the device hosting the parallel processing unit) one or more control messages to connected network devices to influence the routing of network packets based on network packet metadata. For example, kernel code may be configured to send (or cause to be sent) messages to one or more network switches that include one or more rules (e.g., priority scheduling rules, routing rules, etc.). Rules may indicate how traffic should be processed based on its associated metadata (e.g., routing, prioritization, treatment, etc.). For example, a rule may indicate that packets including a first tag in their metadata should have a higher priority than packets including a second tag in their metadata.

[0023] In some embodiments, kernel code or other entities executing it may be configured to respond to queries from connected network devices regarding how specific traffic should be handled (e.g., routed, prioritized, treated, etc.). For example, during kernel code execution, a host processor (e.g., in a system hosting a GPU executing the kernel code) may expose one or more application programming interfaces (APIs). When a network device receives network traffic from the host processor, it can query the APIs to determine how to handle (e.g., prioritized, routed, treated, etc.) the traffic from the host processor. Responses to queries may include one or more rules indicating how the traffic should be handled (e.g., routed, prioritized, treated, etc.) based on associated metadata.

[0024] Therefore, network packets generated for the AI ​​kernel may include information (e.g., metadata, headers, etc.) indicating one or more attributes of the data included in the network packet. As the network packet traverses the network of connected parallel processing units, it can be processed based on the network packet's metadata. In some embodiments, the host processor may provide control messages to network devices (e.g., switches, smart NICs, etc.) to instruct how AI traffic should be processed. In some embodiments, the network device can query the host processor to determine how traffic from one or more parallel processing units of the host should be processed.

[0025] In some embodiments, more than one traffic processing method may be used. For example, when the execution of the AI ​​kernel begins, one or more control messages may be provided to the network device to provide an initial or default set of rules for packet processing (e.g., prioritization, routing, buffering, etc.). Then, during the execution of the AI ​​kernel, if the network device receives a network packet with metadata that does not correspond to any rules stored in the network device, the network device may query the host processor to determine how to process the traffic with the new metadata.

[0026] Compared with the prior art, the advantages of the disclosed embodiments include, but are not limited to, improved network routing of AI traffic, thereby making more efficient use of the computing resources of parallel processing units.

[0027] Figure 1This is a block diagram of an exemplary distributed artificial intelligence (AI) network 100 for aggregate-aware processing of AI model traffic according to at least one embodiment. The distributed AI network 100 may include one or more AI systems, such as AI system 102a, AI system 102b, and AI system 102c connected via one or more AI metadata-aware switches, such as AI metadata-aware switch 104a and AI metadata-aware switch 104b. In some embodiments, the distributed AI network 100 may be part of a data center. In some embodiments, the distributed AI network 100 may be part of a remote platform, such as a cloud-based platform.

[0028] Each AI system may include various components for performing AI workloads. For example, an AI system may include: a host processor (e.g., a central processing unit (CPU)), such as host processor 106a, host processor 106b, or host processor 106c; a parallel processing unit (e.g., a graphics processing unit (GPU)), such as parallel processing unit 108a, parallel processing unit 108b, or parallel processing unit 108c; and an AI metadata-aware network interface controller (NIC), such as AI metadata-aware NIC 110a, AI metadata-aware NIC 110b, or AI metadata-aware NIC 110c. In some embodiments, one or more AI systems of the distributed AI network 100 may include one or more physical machines (e.g., one or more server computers, desktop computers, mobile devices, etc.), one or more virtual machines, and / or one or more containers running on host devices.

[0029] The AI ​​systems of the distributed AI network 100 can collaborate to perform AI workloads. For example, AI computation can be broken down into smaller, not necessarily uniform, parts that are executed in parallel. This computation can be performed using the AI ​​systems of the distributed AI network 100. More specifically, the computation can be performed using the parallel processing units of the AI ​​systems of the distributed AI network 100.

[0030] A process running on a first parallel processing unit (e.g., parallel processing unit 108a) may depend on a process running on one or more other processors (e.g., parallel processing unit 108b and / or parallel processing unit 108c). The parallel processing units may use network packets to transmit interdependent data and / or computation results via AI metadata-aware switches 104a and / or 104b. The coordinated communication and operation between processors can be referred to as a “setup.”

[0031] The AI ​​workload performed may include one or more AI kernels executing on parallel processing units of the distributed AI network 100. AI kernels may be compiled from source code for execution on parallel processing units. Kernel developers who write the source code may designate specific regions of the code as "high priority." When the code is compiled and executed using parallel processing units, network packets created from high-priority code regions and used for transmission to other parallel processing units may be marked as high priority. In some embodiments, one or more headers of the network packets may be modified, or headers may be added, to mark packets as high priority.

[0032] In some embodiments, network packets generated from different regions of the code may be tagged to identify one or more attributes of the data carried by the network packets. This tagging may include one or more labels, flags, or identifiers included in the metadata of the network packet. The tagging may indicate attributes of the network packet and / or its data, such as priority, type of set operation, communication set identifier, data type, set data volume indicator, etc. In some embodiments, the type of set operation may include at least one of broadcast operation, full reduction operation, reduction operation, full collection operation, collection operation, distribution operation, reduction-distribution operation, or barrier operation.

[0033] In some embodiments, tags and / or metadata may indicate an association with one or more parallel processing units of the distributed AI network 100. For example, when multiple parallel processing units are included in a set operation, a communication set identifier may be assigned to the group of parallel processing units. The communication set identifier may be included in network packet metadata to associate the network packet with the group of parallel processing units, and the network packet may be processed (e.g., routed, prioritized, peer-to-peer) based on the communication set identifier. In some embodiments, the network packet may have an associated target parallel processing unit. For example, the network packet may be generated by a first parallel processing unit of the distributed AI network 100. The network packet may include data required by a second parallel processing unit of the distributed AI network 100. A network packet from the first parallel processing unit may identify the second parallel processing unit as a target parallel processing unit among the multiple parallel processing units of the distributed AI network 100.

[0034] Network packets generated by a first code region may be marked as first metadata, which indicates the type of data transmitted by the first code region. Network packets generated by a second code region may be marked as second metadata, which indicates the type of data transmitted by the second code region.

[0035] In some embodiments, an AI kernel executing on a parallel processing unit may send one or more signals to a connected NIC to cause the NIC to generate network packets for the AI ​​kernel. The signals may include metadata to be included in the generated network packets. The signals may also include additional data for the generated network packets, such as data or payload to be included in the packets, the destination address of the packets, etc. In some embodiments, the AI ​​kernel may create network packets and provide them to the NIC for forwarding to its destination.

[0036] When network packets are being processed by network devices (e.g., NICs such as AI metadata-aware NIC 110a, AI metadata-aware NIC 110b, and AI metadata-aware NIC 110c, routers, switches such as AI metadata-aware switch 104a and AI metadata-aware switch 104b), packets identified as high-priority may be given priority. For example, high-priority network packets may be processed before other network packets in the network queue. High-priority network packets may also be routed based on their priority. In some embodiments, network packets may be treated differently based on their metadata. For example, if a particular router or switch in the network becomes congested, non-high-priority packets may be dropped or delayed to reduce congestion and allow high-priority packets to be transmitted.

[0037] In some embodiments, if a particular router or switch in the network becomes congested, packets marked as belonging to a low data volume set may be buffered instead of dropped to avoid retransmission. For example, some communication sets (e.g., groups of associated distributed processors) may generate fewer network packets than other communication sets. Network packets generated by a low data volume communication set may include a set data volume indicator indicating the number of network packets generated by that associated communication set. In some embodiments, the set data volume indicator is a value from a predetermined set of values ​​(e.g., "low", "medium", "high", etc.). In some embodiments, the set data volume indicator is a counter indicating the number of network packets generated by the communication set. In some embodiments, the set data volume indicator is a value indicating the maximum number of network packets that can be generated by the communication set.

[0038] If a network packet is marked as belonging to a low data volume set (for example, if a set data volume indicator indicates that the number of network packets generated by the communication set is less than a first data volume threshold), the router or switch can buffer these packets (because they are not many) and continue processing other packets.

[0039] In some embodiments, when the AI ​​kernel source code is compiled and executed using a parallel processing unit, it may send or cause one or more control messages to be sent to a connected network device to influence the processing of network packets (e.g., routing, buffering, etc.) based on the metadata of the network packets. For example, a parallel processing unit (e.g., parallel processing unit 108a) may generate one or more control messages and provide them to a connected NIC (e.g., AI metadata-aware NIC 110a) for forwarding to a specific network device (e.g., AI data-aware switch 104a). In some cases, the parallel processing unit may provide one or more signals to a connected NIC to cause that NIC to generate control messages and forward them to a network device. In some cases, a process executing on the host processor of the AI ​​system (e.g., host processor 106a of AI system 102a) may generate control messages and provide them to a connected NIC (e.g., AI metadata-aware NIC 110a) for forwarding to a network device. In some cases, a process executing on the host processor of the AI ​​system may provide one or more signals to a connected NIC to cause that NIC to generate control messages and forward them to a network device.

[0040] In some embodiments, the control message may be sent directly to the network device. In some embodiments, the control message may be provided to a central server. For example, the central server may have an application programming interface (API) configured to receive control messages. Before providing the control message (or an alternative control message based on the first control message) to the network device, the central server may perform one or more operations based on the control message, such as aggregation, deduplication, etc.

[0041] In some embodiments, the central server may be located in a data center. In some embodiments, the central server may be part of a remote platform, such as a cloud-based platform.

[0042] Control messages may include one or more rules (e.g., priority scheduling rules, routing rules, buffering rules, etc.) that indicate how traffic should be processed based on its associated metadata (e.g., routing, prioritization, treatment, buffering, etc.). For example, the rule could indicate that packets with a first tag in their metadata should have a higher priority than packets with a second tag in their metadata.

[0043] AI metadata-aware switches 104a and 104b may store one or more rules for processing AI traffic in the distributed AI network 100. When a network packet is received, the metadata associated with that packet can be determined. For example, one or more headers of the network packet can be identified to determine the metadata of the network packet. The AI ​​metadata-aware switches 104a and / or 104b may then process the network packet based on the determined metadata. For example, the AI ​​metadata-aware switch 104a may include rules indicating that network packets containing a first tag in their metadata should be given high priority.

[0044] In some embodiments, the AI ​​kernel, or other entities executing it on its behalf (such as the host processor of the AI ​​system executing the AI ​​kernel), may be configured to respond to queries from connected network devices regarding how specific traffic should be handled. For example, during kernel code execution, the host processor of the AI ​​system may expose one or more APIs. When a network device receives network traffic from the AI ​​system, it may query these APIs to determine how to handle (e.g., prioritize, route, buffer, treat, etc.) the traffic from the AI ​​system. The response to this query may include one or more rules instructing how the traffic should be handled based on its associated metadata.

[0045] In some embodiments, more than one traffic processing method may be used. For example, when the execution of the AI ​​kernel begins, one or more control messages may be provided to the network device to provide an initial or default set of rules for packet processing (e.g., prioritization, routing, buffering, etc.). Then, during the execution of the AI ​​kernel, if the network device receives a network packet with metadata that does not correspond to any rules stored in the network device, the network device may query the AI ​​system to determine how to process the traffic with the new metadata.

[0046] Figure 2This is a block diagram of an exemplary artificial intelligence (AI) system 202 for aggregate-aware processing of AI model traffic according to at least one embodiment. The AI ​​system 202 may be included in a network of AI systems for performing distributed AI operations. The AI ​​system 202 may include a host processor 204, a parallel processing unit 206, and an AI metadata-aware NIC 210. The host processor 204 may include one or more central processing units (CPUs), accelerators, physical processing units (PPUs), data processing units (DPUs), and / or other processing circuitry systems. The parallel processing unit 206 may include one or more graphics processing units (GPUs). In some embodiments, the AI ​​system 202 may include memory (not shown), such as read-only memory (ROM), flash memory, dynamic random access memory (DRAM, such as synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or DRAM (RDRAM), etc.).

[0047] The host processor 204 may execute one or more processes to coordinate distributed AI operations. For example, in some embodiments, the host processor 204 may execute AI traffic rules API 214 in response to queries from network devices (e.g., routers, switches, etc.) that inquire how specific AI traffic should be processed (e.g., routing, prioritizing, buffering, treating, etc.).

[0048] Parallel processing unit 206 may include one or more processors, such as a graphics processing unit (GPU), for parallel execution. Parallel processing unit 206 may execute AI kernel 208, which may include one or more distributed AI operations. In some embodiments, AI kernel 208 includes one or more AI models and / or machine learning models. In some embodiments, AI kernel 208 is configured to train AI models, such as large language models (LLMs), visual models, visual language models, etc. In some embodiments, AI kernel 208 is configured to perform inference on the trained AI models. In some embodiments, AI kernel 208 is configured to perform inference about... Figure 9A , Figure 9B and / or Figure 9C The one or more models mentioned above.

[0049] To perform distributed AI operations, AI kernel 208 may generate (or enable AI metadata-aware NIC 210 to generate) one or more network packets with metadata associated with AI kernel 208. For example, network packets may include one or more modified headers or one or more additional headers that may include information related to the AI ​​operation being performed or information related to the data included in the network packet. For example, metadata may include one or more tags, flags, or identifiers and may indicate attributes of the network packet and / or its data, such as priority, type of set operation, communication set identifier, data type, set data volume indicator, etc. In some embodiments, parallel processing unit 206 may execute multiple AI kernels simultaneously, each AI kernel generating its own AI network traffic.

[0050] The AI ​​metadata-aware NIC 210 may include a network interface controller (NIC) configured to process traffic based on AI metadata included in network packets. For example, the AI ​​metadata-aware NIC 210 may include one or more queues or buffers for network packets. When network packets are generated, they may be stored in queues or buffers before being transmitted over the network. Based on the AI ​​metadata included in the network packets, some network packets may be removed from the queues or buffers and transmitted before other network packets. For example, a network packet with metadata indicating high priority may be transmitted before a network packet with metadata not indicating high priority.

[0051] In some embodiments, the AI ​​metadata-aware NIC 210 may be a “smart NIC” that may include a processor 212, independent of the host processor 204, for performing one or more operations. In some embodiments, the processor 212 is a data processing unit (DPU). In some embodiments, the processor 212 may execute an AI traffic rules API 216 in response to a query from a network device (e.g., a router, switch, etc.) that inquires how specific AI traffic should be processed (e.g., routed, prioritized, buffered, etc.).

[0052] Figure 3 This is a block diagram of an exemplary artificial intelligence (AI) metadata-aware switch 302 for aggregate-aware processing of AI model traffic according to at least one embodiment. The AI ​​metadata-aware switch 302 can receive AI model traffic in a distributed AI network and can process the traffic based on metadata included in network packets.

[0053] The AI ​​metadata-aware switch 302 may include a processing unit 304 and a memory 306. The processing unit 304 may include one or more central processing units (CPUs), graphics processing units (GPUs), accelerators, physical processing units (PPUs), data processing units (DPUs), application-specific integrated circuits (ASICs), etc. The memory 306 may include read-only memory (ROM), flash memory, dynamic random access memory (DRAM, such as synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or DRAM (RDRAM), etc.).

[0054] Processing unit 304 can be configured to receive and execute AI traffic processing rules. In some embodiments, the rules are received from a connected AI system, such as... Figure 4A As described. In some embodiments, processing unit 304 may generate a query to inquire how the connected AI system should handle specific network traffic. AI metadata-aware switch 302 may receive a response to the query, and the response may include one or more rules.

[0055] The memory 306 may include one or more rules 308 for instructing how AI network traffic should be processed. The rule may include one or more metadata values ​​for matching network packets against the rule, and may include one or more actions or operations to be performed on the matched network packets. In some embodiments, the rule may indicate the priority of each network packet. In some embodiments, the rule may indicate the route that should be used for a specific subset of the AI ​​network traffic.

[0056] Figure 4A This is an example timing diagram 402 showing an artificial intelligence (AI) system providing rules to an AI metadata-aware switch according to at least one embodiment. The AI ​​system 404 and the AI ​​metadata-aware switch 406 may be included in a distributed AI network (such as...). Figure 1 In a distributed AI network 100, the AI ​​system 404 can generate one or more network packets during AI kernel execution. These network packets can be tagged with metadata related to the operation being performed. For example, some network packets may include a communication set identifier or a set operation type. An AI metadata-aware switch 406 can be configured to process network packets based on their metadata.

[0057] In some embodiments, AI system 404 may provide one or more rules to AI metadata-aware switch 406 to indicate how AI network traffic should be processed. For example, AI system 404 may provide rule 408a to AI metadata-aware switch 406. Rule 408a may include one or more rules for processing AI network traffic (e.g., routing, prioritizing, buffering, treating, etc.). Rules may include one or more metadata values ​​for matching network packets against the rules, and may include one or more actions or operations to be performed on the matched network packets. In some embodiments, rules may indicate the priority of each network packet. In some embodiments, rules may indicate the routing that should be used for a specific subset of AI network traffic.

[0058] In some embodiments, rule 408a may include all rules for network traffic from AI system 404. In some embodiments, AI system 404 sends a first set of rules in rule 408a, followed by one or more sets of rules in rules 408b and / or rule 408c. In some cases, rule 408a may include a first rule for a first metadata label, and rule 408b may include a second rule for the first metadata label, which overrides the first rule.

[0059] In some embodiments, rule 408a is generated by the parallel processing unit of AI system 404. In some embodiments, rule 408a is generated by the host processor of AI system 404. In some embodiments, rule 408a is generated by the NIC of AI system 404.

[0060] Figure 4B This is an example timing diagram 410 showing an artificial intelligence (AI) system responding to a query from an AI metadata-aware switch according to at least one embodiment. The AI ​​system 412 and the AI ​​metadata-aware switch 414 may be included in a distributed AI network (such as...). Figure 1 In a distributed AI network 100, the AI ​​system 412 can generate one or more network packets during the execution of the AI ​​kernel. These network packets may be tagged with metadata related to the operation being performed. For example, some network packets may include a communication set identifier or a set operation type. An AI metadata-aware switch 414 can be configured to process network packets based on their metadata.

[0061] If the AI ​​metadata-aware switch 414 receives AI network traffic whose metadata does not match any rules stored at the AI ​​metadata-aware switch 414, the AI ​​metadata-aware switch 414 may send a query 416a to the AI ​​system 412. Query 416a may include a request for one or more rules that match a specific metadata value. The AI ​​system 412 may respond to query 416a with a response 418a, which may include one or more rules for processing the AI ​​network traffic. For example, response 418a may include one or more rules that match the metadata value included in query 416a. After receiving response 418a, the AI ​​metadata-aware switch 414 may process the network traffic based on the rules included in response 418a.

[0062] In some embodiments, the AI ​​metadata-aware switch 414 may receive second network traffic at a second time, the metadata of which does not match any rules stored at the AI ​​metadata-aware switch 414. The AI ​​metadata-aware switch 414 may send a query 416b to the AI ​​system 412 and may receive a response 418b having one or more rules that match the metadata values ​​included in the query 416b. In some embodiments, the response (e.g., response 418a, response 418b, etc.) may include more than one rule. In some embodiments, the response may include one or more rules that do not match the metadata values ​​included in the corresponding query. For example, the AI ​​metadata-aware switch 414 may query the AI ​​system 412 about a first metadata tag, and the AI ​​system 412 may respond using a first rule for metadata with the first metadata tag and a second rule for metadata with a second metadata tag.

[0063] In some embodiments, response 418a is generated by a parallel processing unit of AI system 412. In some embodiments, response 418a is generated by a host processor of AI system 412. For example, the host processor of AI system 412 may provide an API for receiving AI traffic queries. In some embodiments, response 418a is generated by a NIC of AI system 412. For example, the NIC of AI system 412 (e.g., a smart NIC, DPU, etc.) may provide an API for receiving AI traffic queries.

[0064] In some embodiments, the AI ​​metadata-aware switch 414 can initiate queries to individual entities that can respond to the queries on behalf of the AI ​​system 412. For example, a central server can be configured to receive and respond to queries on behalf of multiple AI kernels executing on one or more parallel processing units.

[0065] Figure 5 This is a flowchart of an example method 500 for aggregate-aware processing of AI model traffic according to at least one embodiment. Figure 6 This is a flowchart of an example method 600 for aggregate-aware processing of AI model traffic according to at least one embodiment.

[0066] Method 500 and / or method 600 may be executed using one or more processing units (e.g., CPU, GPU, accelerator, physical processing unit (PPU), data processing unit (DPU), etc.), which may include one or more memory devices (or communicate with one or more memory devices). In at least one embodiment, method 500 and / or method 600 may be executed using one or more processing devices. In at least one embodiment, method 500 and / or method 600 may be executed using one or more hardware circuits. In some embodiments, the hardware circuits may interact with one or more software processes. In at least one embodiment, method 500 may use... Figure 1 The processing unit of the AI ​​metadata-aware NIC 110a is executed. In at least one embodiment, method 600 may be performed by... Figure 1 The AI ​​metadata-aware switch 104a executes. In at least one embodiment, the processing unit for executing any one of methods 500 and / or 600 can execute instructions stored on a non-transitory computer-readable storage medium. In at least one embodiment, any one of methods 500 and / or 600 can be executed using multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads for implementing any one of methods 500 and / or 600 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads for implementing any one of methods 500 and / or 600 can execute asynchronously relative to each other. Figure 5 and / or Figure 6 Compared to the order shown, the various operations of method 500 and / or method 600 can be performed in a different order. Some operations of any of the methods in method 500 and / or method 600 can be performed concurrently with other operations. In at least one embodiment, Figure 5 and / or Figure 6 One or more of the operations shown may not always be performed.

[0067] Now refer to Figure 5At block 502, the processing unit performing method 500 may generate a first network packet having first metadata. The first metadata may be determined based on code used to create the first network packet and / or based on data included in the first network packet. The first metadata may indicate a first association with a plurality of parallel processing units for performing distributed artificial intelligence (AI) operations. In some embodiments, the first association includes one or more target parallel processing units among the plurality of parallel processing units. In some embodiments, the distributed AI operation includes one or more of a broadcast operation, a full reduction operation, a reduction operation, a full collection operation, a collection operation, a distribution operation, a reduction-distribution operation, or a barrier operation.

[0068] In some embodiments, a first network packet having associated first metadata is generated for an AI kernel executed using a first parallel processing unit among a plurality of parallel processing units. In some embodiments, the first network packet is generated by the first parallel processing unit. In some embodiments, the first network packet is generated by a host processor associated with the first parallel processing unit. In some embodiments, the first network packet is generated by a NIC associated with the first parallel processing unit.

[0069] In some embodiments, the first metadata includes at least one of a first communication set identifier, a data type, or an operation type. In some embodiments, the operation type is a set operation type.

[0070] At block 504, after generating the first network packet, the processing unit may generate a second network packet having second metadata. The second metadata may be determined based on the code used to create the second network packet and / or based on the data included in the second network packet. In some embodiments, the first metadata includes a first communication set identifier, and the second metadata includes a second communication set identifier that is different from the first communication set identifier.

[0071] At block 506, the processing unit may determine that the second network packet has a higher priority than the first network packet based on a comparison of the second metadata and the first metadata (e.g., one or more rules may indicate that a network packet with metadata included in the second metadata should take precedence over a network packet with metadata included in the first metadata, the priority value included in the second metadata being higher than the priority value in the first metadata, etc.). At block 508, the processing unit may provide the second network packet to the networking device before providing the first network packet. In some embodiments, the networking device is a network switch configured to process network traffic based on AI-related metadata.

[0072] Now refer to Figure 6At block 602, the processing unit performing method 600 may receive a first network data packet. At block 604, after receiving the first network data packet, the processing unit may receive a second network data packet. At block 606, the processing unit may determine a first rule for processing the first network data packet based on first metadata associated with the first network data packet, and a second rule for processing the second network data packet based on second metadata associated with the second network data packet. In some embodiments, the rules may be determined in conjunction with the following discussion in block 612. In some embodiments, information received from a second networking device may be used to determine the rules, as discussed below in conjunction with blocks 614 and 616.

[0073] The first metadata may indicate a first association with multiple parallel processing units used to perform distributed artificial intelligence (AI) operations. The second metadata may indicate a second association with multiple parallel processing units used to perform distributed AI operations.

[0074] In some embodiments, the first metadata includes at least one of a first communication set identifier, a data type, or an operation type. In some embodiments, the operation type is a set operation type.

[0075] In some embodiments, at block 612, in order to determine a first rule for processing a first network packet and a second rule for processing a second network packet, the processing unit may receive one or more control messages, the control messages including the first rule for processing the first network packet and the second rule for processing the second network packet. In some embodiments, the one or more control messages are generated for an AI kernel executed using a first parallel processing unit among a plurality of parallel processing units.

[0076] In some embodiments, at block 614, to determine a first rule for processing a first network packet and a second rule for processing a second network packet, the processing unit may send a query to a second network device. At block 616, the processing unit may, in response to the query, receive a response from the second network device, the response including the first rule for processing the first network packet and the second rule for processing the second network packet. In some embodiments, the response is generated for an AI kernel executed by a first parallel processing unit among a plurality of parallel processing units.

[0077] In some embodiments, the first rule for processing the first network packet, determined based on the first metadata associated with the first network packet, includes at least one of a priority scheduling rule for the first network packet, a buffering decision for the first network packet, or a network routing rule for the first network packet.

[0078] At block 608, the processing unit may determine that the second network data packet has a higher priority than the first network data packet based on a comparison of the second metadata and the first metadata. At block 610, the processing unit may provide the second network data packet to the first networking device before providing the first network data packet.

[0079] Computer Architecture Figure 7 This is a block diagram illustrating an exemplary computer system 700 according to at least one embodiment. The exemplary computer system may be a system with interconnected devices and components, a system-on-a-chip (SoC), or some combination thereof formed with a processor, which may include an execution unit to execute instructions. In at least one embodiment, according to this disclosure, such as the embodiments described herein, the computer system 700 may include, but is not limited to, components such as a processor 702 equipped with an execution unit including logic to execute algorithms for processing data. In at least one embodiment, the computer system 700 may include a processor, such as a PENTIUM® family processor, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessor available from Intel Corporation of Santa Clara, California, although other systems (including PCs, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, the computer system 700 may execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0080] The embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (IP) devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip (SoC), a network computer (“NetPC”), a set-top box, a network hub, a wide area network (“WAN”) switch, an edge device, an Internet of Things (“IoT”) device, or any other system that can execute one or more instructions according to at least one embodiment.

[0081] In at least one embodiment, the computer system 700 may include, but is not limited to, a processor 702, which may include, but is not limited to, one or more execution units 708, to perform machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 700 is a single-processor desktop or server system, but in another embodiment, the computer system 700 may be a multiprocessor system. In at least one embodiment, the processor 702 may include, but is not limited to, a Complex Instruction Set Computer (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing instruction set combinations, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 702 may be coupled to a processor bus 710, which enables the transfer of data signals between the processor 702 and other components in the computer system 700.

[0082] In at least one embodiment, processor 702 may include, but is not limited to, a Level 1 (“L1”) internal cache memory (“cache”) 704. In at least one embodiment, processor 702 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside outside of processor 702. Depending on specific implementation and requirements, other embodiments may also include a combination of internal and external caches.

[0083] In at least one embodiment, processor 702 may include, but is not limited to, a Level 2 (“L2”) internal cache memory (“cache”) 704. The L2 cache, compared to the L1 cache, serves as a larger and slightly slower secondary cache, but its access speed is still faster than accessing main memory (e.g., via memory controller hub 716). Therefore, the L2 cache can improve performance by reducing the time spent by the processor accessing main memory. In at least one embodiment, processor 702 may have a single internal L2 cache or multiple levels of internal cache. In embodiments where processor 702 is a multi-core processor, the L2 cache may be shared among multiple cores of processor 702, providing a larger intermediate level of cache memory for more than one processing core. In at least one embodiment, the L2 cache memory may reside externally to processor 702.

[0084] In at least one embodiment, processor 702 may include, but is not limited to, a Level 3 (“L3”) internal cache memory (“cache”) 704. Compared to L1 and L2 caches, the L3 cache can serve as a larger and slower Level 3 cache. The L3 cache can improve performance by reducing the time the processor spends accessing main memory. The L3 cache can be shared among multiple cores of processor 702, providing a larger pool of fast-access memory for the data of each processor core. In at least one embodiment, processor 702 may have a single internal L3 cache or multiple levels of internal caches. In at least one embodiment, the L3 cache memory may reside outside of processor 702. Depending on specific implementation and requirements, other embodiments may also include any combination of internal or external L1, L2, and / or L3 caches. In at least one embodiment, register file 706 may store different types of data in various registers, including but not limited to integer registers, floating-point registers, status registers, and instruction pointer registers.

[0085] In at least one embodiment, execution unit 708 (including, but not limited to, logic for performing integer and floating-point operations) also resides in processor 702. In at least one embodiment, processor 702 may also include microcode (“ucode”) read-only memory (“ROM”) storing microcode of certain macro instructions. In at least one embodiment, execution unit 708 may include logic for processing packaged instruction set 709. In at least one embodiment, by including packaged instruction set 709 in the instruction set of general-purpose processor 702, along with the associated circuitry for executing the instructions, packaged data in general-purpose processor 702 can be used to perform operations used by many multimedia applications. In one or more embodiments, operations on packaged data can be performed using the full width of the processor's data bus, thereby accelerating and more efficiently executing many multimedia applications, which may eliminate the need to transfer smaller data units on the processor's data bus to perform one or more operations on one data element at a time.

[0086] In at least one embodiment, execution unit 708 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuitry. In at least one embodiment, computer system 700 may include, but is not limited to, memory 720. In at least one embodiment, memory 720 may be implemented as a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or other memory device. In at least one embodiment, memory 720 may store one or more instructions 719 and / or data 721 represented by data signals executable by processor 702.

[0087] In at least one embodiment, the system logic chip may be coupled to the processor bus 710 and the memory 720. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 716, and the processor 702 may communicate with the MCH 716 via the processor bus 710. In at least one embodiment, the MCH 716 may provide a high-bandwidth memory path 718 to the memory 720 for instruction and data storage, as well as for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 716 may direct data signals between the processor 702, the memory 720, and other components in the computer system 700, and bridge data signals between the processor bus 710, the memory 720, and the system I / O 722. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH 716 can be coupled to memory 720 via high-bandwidth memory path 718, and graphics / video card 712 can be coupled to MCH 716 via Accelerated Graphics Port (“AGP”) interconnect 714.

[0088] In at least one embodiment, computer system 700 may use system I / O 722 as a dedicated hub interface bus to couple MCH 716 to I / O controller hub (“ICH”) 730. In at least one embodiment, ICH 730 may provide direct connectivity to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to memory 720, chipset, and processor 702. Examples may include, but are not limited to, an audio controller 729, a firmware hub (“Flash BIOS”) 728, a wireless transceiver 726, a data storage device 724, a conventional I / O controller 723 including a user input and keyboard interface 725, a serial expansion port 727 (such as Universal Serial Bus (USB)), and a network controller 734, which in some embodiments may include a data processing unit. Data storage device 724 may include a hard disk drive, floppy disk drive, CD-ROM device, flash memory device, or other mass storage device.

[0089] In at least one embodiment, Figure 7 A system including interconnected hardware devices or "chips" is shown, while in other embodiments, Figure 7An exemplary system-on-a-chip (“SoC”) may be illustrated. In at least one embodiment, the device may be interconnected with a dedicated interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of the computer system 700 are interconnected using a compute fast link (CXL) interconnect.

[0090] Inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 715 may be... Figure 7 Used in systems for reasoning or predicting operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures or neural network use cases described herein.

[0091] These components can be used to generate synthetic data that mimics failure cases during network training, which can help improve the network's performance while limiting the amount of synthetic data to avoid overfitting.

[0092] Figure 8 This is a block diagram illustrating an electronic device 800 for utilizing a processor 810 according to at least one embodiment. In at least one embodiment, the electronic device 800 may be, for example, but not limited to, a laptop computer, tower server, rack server, blade server, desktop computer, tablet computer, mobile device, telephone, embedded computer, edge device, IoT device, or any other suitable electronic device.

[0093] In at least one embodiment, electronic device 800 may include, but is not limited to, processor 810, which is communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, processor 810 is coupled using a bus or interface, such as an I2C bus, a system management bus (“SMBus”), a low pin count (LPC) bus, a serial peripheral interface (“SPI”), a high-definition audio (“HDA”) bus, a serial advanced technology accessory (“SATA”) bus, a universal serial bus (“USB”) (versions 1, 2, and 3), or a universal asynchronous receiver / transmitter (“UART”) bus. In at least one embodiment, Figure 8 A system including interconnected hardware devices or "chips" is shown, while in other embodiments, Figure 8 An exemplary system-on-a-chip (“SoC”) may be illustrated. In at least one embodiment, Figure 8 The device shown can be interconnected with a dedicated interconnect, a standardized interconnect (such as PCIe), or some combination thereof. In at least one embodiment, Figure 8 One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0094] In at least one embodiment, Figure 8 This may include a display 824, a touchscreen 825, a touchpad 830, a near-field communication (“NFC”) unit 845, a sensor hub 840, a thermal sensor 846, a fast chipset (“EC”) 835, a trusted platform module (“TPM”) 838, a BIOS / firmware / flash memory (“BIOS, FW flash”) 822, a DSP 860, a drive 820 (such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network (“WLAN”) unit 850, a Bluetooth unit 852, a wireless wide area network (“WWAN”) unit 856, a global positioning system (GPS) 855, a camera (“USB 3.0 camera”) 854 (such as a USB 3.0 camera), and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 815 implemented in, for example, the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0095] In at least one embodiment, other components may be communicatively coupled to processor 810 via the components discussed above. In at least one embodiment, accelerometer 841, ambient light sensor (“ALS”) 842, compass 843, and gyroscope 844 may be communicatively coupled to sensor hub 840. In at least one embodiment, thermal sensor 839, fan 837, keyboard 836, and touchpad 830 may be communicatively coupled to EC 835. In at least one embodiment, speaker 863, earphone 864, and microphone (“MIC”) 865 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 862, which in turn may be communicatively coupled to DSP 860. In at least one embodiment, audio unit 862 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 857 may be communicatively coupled to WWAN unit 856. In at least one embodiment, components such as WLAN unit 850, Bluetooth unit 852, and WWAN unit 856 may be implemented in a next-generation form factor (NGFF).

[0096] In at least some embodiments, language models, such as Large Language Models (LLMs) and / or other types of generative artificial intelligence (AI), may be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, omniverse and / or metaverse document information (e.g., in USD format), etc., based on context provided in input prompts or queries. In embodiments, these language models may be considered “large” based on the fact that they are trained on massive datasets and have architectures that include a large number of learnable network parameters (weights and biases)—such as millions or billions of parameters. LLMs, VLMs, etc., can be used to summarize textual data, analyze data (e.g., text, images, videos, etc.) and extract insights from the data, and generate new text, images, videos, etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM of this disclosure may be specifically designed for text processing, while in others a multimodal LLM may be implemented for receiving, understanding, and / or generating text and other types of content, such as images, audio, and / or video. For example, a visual language model (VLM) or a generalized multimodal language model may be implemented for receiving images, video, audio, text, 3D designs (e.g., CAD), and / or other input data types, and / or for generating or outputting images, video, audio, text, 3D designs, and / or other output data types.

[0097] In various embodiments, various types of LLM / VLM architectures can be implemented. For example, different architectures can be implemented that use different techniques to understand and generate outputs (such as text, audio, video, images, etc.). In some embodiments, LLM architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can be used; while in other embodiments, transformer architectures (such as architectures relying on self-attention mechanisms) can be used to understand and recognize relationships between words or tokens. One or more generative processing pipelines including an LLM may also include one or more diffusion blocks (e.g., denoisers). The language model of this disclosure may include one or more encoder and / or decoder blocks. For example, discriminative LLMs or pure encoder LLMs such as BERT (transformer-based bidirectional encoder representation) can be implemented for tasks involving language understanding (such as classification, sentiment analysis, question answering, and named entity recognition). As another example, generative LLMs or pure decoder LLMs such as GPT (generative pre-trained transformer) can be implemented for tasks involving language and content generation (such as text completion, story generation, and dialogue generation). An LLM that includes both encoder and decoder components (like T5 (text-to-text transformer)) can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting; any architecture type (including, but not limited to, those described herein) can be implemented according to a particular embodiment and using one or more models to perform one or more tasks.

[0098] In various embodiments, LLMs / VLMs, etc., can be trained using unsupervised learning, where the LLM learns patterns from large amounts of unlabeled text / audio / video / image data, etc. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM extensively pre-trained on massive amounts of unlabeled text data can be referred to as a base model and can excel at various tasks such as question answering, summarizing, filling in missing information, and translation. Some LLMs can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or lexical units to bias the language model towards a specific task or domain), and / or using other fine-tuning or customization techniques for optimizing the model on a specific task and / or within a specific domain.

[0099] In some embodiments, the LLM / VLM of this disclosure may be implemented using various model alignment techniques. For example, in some embodiments, guardrails may be implemented to identify inappropriate or undesirable inputs (e.g., prompts) and / or outputs of the model. In some non-limiting embodiments, the implemented guardrails may be similar to those described in U.S. Patent Application No. 18,304,341, filed April 20, 2023, the entire contents of which are incorporated herein by reference. In some embodiments, one or more additional models or layers thereof may be implemented to identify problems with the model's inputs and / or outputs. For example, these "guardrail" models may be trained to identify inputs and / or outputs that are "safe" or otherwise feasible or desired for a particular application / implementation and / or "unsafe" or otherwise undesirable. Thus, the LLM / VLM of this disclosure may be less likely to output language / text / audio, etc., that is offensive, vulgar, inappropriate, unsafe, extraterritorial, and / or otherwise undesirable for a particular application / implementation.

[0100] In some embodiments, LLM / VLM, etc., may be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt) to access one or more plugins (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt relates to restaurants or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. As another example, where at least part of the response requires mathematical computation, the model may access one or more mathematical plugins or APIs to help solve one or more problems, and then use the responses from the plugins and / or APIs for the model's output. This process can be repeated for any number of iterations, using any number of plugins and / or APIs—e.g., recursively—until a response to the input prompt is generated to resolve each query / question / request / process / operation, etc. Therefore, one or more models can rely not only on the knowledge they gain from training on one or more large datasets, but also on the expertise or optimization properties of one or more external resources, such as APIs, plugins, etc.

[0101] In some embodiments, multiple language models (e.g., LLM / VLM, multiple instances of the same language model, and / or multiple prompts provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and prompts (e.g., a set of constraints, various conditions, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one prompt to constrain, guide, or otherwise influence the style, content, or persona of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, persons, or different knowledge bases, all of which are defined by the provided prompts.

[0102] In any such embodiment, the output of objects such as: two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated agents of at least one language model, and / or two or more prompts provided to at least one language model, wherein the processing is, for example, summarizing, comparing, or filtering, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or agent) may be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model may be required to generate or otherwise obtain output about the input source material, which is associated with the input source material. Such association may include, for example, generating captions or portions of text and embedding them (e.g., as metadata) in the input source text or image. In one or more embodiments, the output of the language model may be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model may be used to evaluate the presence of a target word in a text portion or the presence of an object in an image, and to annotate the text or image to record this presence (or absence). Alternatively, for example but not limited to, the determination of the language model can be used to determine whether source material should be included in a filtered dataset.

[0103] Figure 9AThis is a block diagram of an example generative language model system 900 applicable to implementing at least some embodiments of the present disclosure. Figure 9A In the example shown, the generative language model system 900 includes a retrieval enhancement generation (RAG) component 992, an input processor 905, a tokenizer 910, an embedding component 920, a plugin / API 995, and a generative language model (LM) 930 (which may include LLM, VLM, multimodal LM, etc.).

[0104] In general, the input processor 905 can receive input 901, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data, etc.), depending on the architecture of the generative LM 930. In some embodiments, input 901 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 901 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In some embodiments where the generative LM 930 is capable of handling multimodal input, input 901 may combine text with image data, audio data, and / or other types of input data (such as, but not limited to, those described herein). Taking raw input text as an example, the input processor 905 can prepare the raw input text in various ways. For example, the input processor 905 can perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words) from relevant text content. In examples involving stop words (common words that tend to carry little semantic meaning), the input processor 905 can remove stop words to reduce noise and allow the generative LM930 to focus on more meaningful content. The input processor 905 can apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases such as abbreviations or acronyms to ensure consistency. These are just a few examples; other types of input processing can be applied.

[0105] In some embodiments, the RAG component 992 can be used to retrieve additional information as part of the input 901 or prompt. For example, in some embodiments, in addition to the data retrieved using the RAG component 992, the input 901 may be generated using queries or inputs to the model (e.g., questions, requests, etc.). In some embodiments, the input processor 905 may analyze the input 901 and communicate with the RAG component 992 (or in some embodiments, the RAG component 992 may be part of the input processor 905) to identify relevant text and / or other data and provide it as additional context or information sources to the generative LM 930, typically from which responses, replies, or outputs 990 are identified. For example, when the input indicates that a user is interested in the desired tire pressure for a particular brand and model of vehicle, the RAG component 992 may use, for example, a vector search in an embedded space, to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits a chatbot related to a specific product offering or service, the RAG component 992 can retrieve previously stored conversation history (or at least its summary) and include the previous conversation history along with the current inquiry / request as part of the input 901 of the generative LM 930.

[0106] Lexicalizer 910 can segment (e.g., processed) text into smaller units (lexicals) for subsequent analysis and processing. Depending on the specific implementation, these lexicals can represent individual words, subwords, characters, etc. Word-based lexicalization divides the text into individual words, treating each word as a separate lexical. Subword lexicalization breaks down words into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 930 to understand morphological changes and process non-lexical words more effectively. Character-based lexicalization represents each character as a separate lexical, allowing the generative LM 930 to process text at a fine-grained level. The choice of lexicalization strategy may depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, lexicalizer 910 can convert (e.g., processed) text into a structured format based on the lexicalization pattern implemented in a particular embodiment.

[0107] Embedding component 920 may use any known embedding technique to transform discrete lexical units into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 920 may use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, term frequency-inverse document frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or other methods.

[0108] In some embodiments where input 901 includes image data, input processor 905 may resize the image data to a standard size compatible with the format of the corresponding input channel, and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 920 may encode the image data using any known techniques (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some embodiments where input 901 includes audio data, input processor 905 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 920 may use any known techniques to extract and encode audio features—such as in the form of spectrograms (e.g., Mel spectrograms). In some embodiments where input 901 includes video data, input processor 905 may extract frames or apply resizing to extracted frames, and embedding component 920 may extract features such as optical flow embedding or video embedding, and / or encode temporal information or frame sequences. In some implementations where input 901 includes multimodal data, the embedding component 920 may use techniques such as early fusion (stitching), late fusion (sequential processing), and attention-based fusion to fuse representations of different types of data (e.g., text, images, audio).

[0109] Other components of the generative LM 930 and / or generative language model system 900 may use different types of neural network architectures depending on the implementation. For example, a transformer-based architecture (such as the one used in models like GPT) may be implemented, which may include: a self-attention mechanism for weighting the importance of different words or lexical units in the input sequence; and / or a feedforward network for processing the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transformers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures combining different types of architectures, adversarial networks for joint distributed learning (such as generative adversarial networks or GANs), or adversarial autoencoders (AAEs), etc. Therefore, depending on the implementation and architecture, the embedded component 920 can apply the encoded representation of the input 901 to the generative LM 930, which can process the encoded representation of the input 901 to generate an output 990, which may include response text and / or other types of data.

[0110] As described herein, in some embodiments, the generative LM 930 may be configured to access or use—or be able to access or use—plugins / APIs 995 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 930 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 992) to access one or more plugins / APIs 995 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 995 to the plugin / API 995, which can process the information and return a response to the generative LM 930, which can use the response to generate output 990. This process can be repeated for any number of iterations, using any number of plugins / APIs 995—for example, recursively—until an output 990 is generated that solves each query / question / request / procedure / action in the input 901. Therefore, one or more models can rely not only on the knowledge they gain from training on one or more large datasets and / or on data retrieved using the RAG component 992, but also on the expertise or optimization properties of one or more external resources, such as APIs / plugins 995.

[0111] Figure 9B This is a block diagram of an exemplary implementation of the generative LM 930, which includes a transformer encoder-decoder. For example, suppose input text such as "who discovered gravity" is lexicalized (e.g., via...). Figure 9A The lexer 910 is used to represent lexical units such as words, then each lexical unit is encoded (e.g., by...). Figure 9A The embedding component 920 is the corresponding embedding (e.g., of size 512). Since these lexical embeddings do not typically represent the position of the lexical in the input sequence, any known technique can be used to add positional encoding to each lexical embedding to encode the order relation and context of the lexical in the input sequence. Thus, (e.g., the resulting) embedding can be applied to one or more encoders 935 of the generative LM 930.

[0112] In an exemplary implementation, one or more encoders 935 form an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In an exemplary transformer architecture, each lexical unit (e.g., a word) flows through a separate path. Thus, each encoder can accept a sequence of vectors, passing each vector sequentially through the self-attention layer, the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each lexical unit (word), a query vector, a key vector, and a value vector can be created for each lexical unit. Pairwise self-attention scores can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoders can apply multi-head attention, where attention mechanisms and different learnable weight matrices are applied in parallel multiple times. Any number of encoders can be cascaded to generate context vectors that encode the input. An attention projection layer 940 can transform the context vectors into attention vectors (keys and values) for one or more decoders 945.

[0113] In an exemplary implementation, one or more decoders 945 form a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer (which uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence), and a feedforward network. Similar to one or more encoders 935, in the exemplary transformer architecture, each lexical unit (e.g., a word) flows through a separate path in one or more decoders 945. During the first pass, one or more decoders 945, a classifier 950, and a generation mechanism 955 generate a first lexical unit, which the generation mechanism 955 can use as input in the second pass. This process is repeated cyclically, successively generating lexical units (e.g., words) and adding them to the output of the previous pass, and in subsequent passes, applying lexical embeddings, including positionally encoded sequences, as input to one or more decoders 945, generating one lexical unit serially each time (called autoregression), until a symbol or lexical unit indicating the end of the response is predicted. Within each decoder, the self-attention layer is typically constrained to focus only on previous positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In an exemplary implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention in one or more encoders 935, except that it creates queries from the layers below it and obtains keys and values ​​(e.g., matrices) from the output of one or more encoders 935.

[0114] Therefore, one or more decoders 945 can output some decoded (e.g., vector) representations of the inputs applied in a particular pass. Classifier 950 can include a multi-class classifier comprising one or more neural network layers that project the decoded (e.g., vector) representations onto corresponding dimensions (e.g., one dimension for each supported word or lexical in the output vocabulary) and convert logit probabilities to probabilities via a softmax operation. Thus, generation mechanism 955 can select or sample words or lexicals based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby sequentially generating each word or lexical. Generation mechanism 955 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or lexical indicating the end of the response is selected or sampled, at which point generation mechanism 955 can output the generated response.

[0115] Figure 9C This is a block diagram of an exemplary implementation of a decoder-only converter architecture in which the generative LM 930 includes a decoder-only converter. For example, Figure 9C One or more decoders 960 can be with Figure 9B One or more decoders 945 operate similarly, with the difference being: Figure 9C Each of the one or more decoders 960 omits the encoder-decoder self-attention layer (because there is no encoder in this embodiment). Therefore, one or more decoders 960 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, symbols or terms indicating the end of the input sequence (or the beginning of the output sequence) are appended to the input sequence, and the resulting sequence (e.g., including positionally encoded corresponding embeddings) is applied to one or more decoders 960. Figure 9B Like one or more decoders 945, each lexical unit (e.g., a word) can flow through a separate path in one or more decoders 960, and one or more decoders 960, classifier 965, and generation mechanism 970 can use autoregression to sequentially generate one lexical unit at a time until a symbol or lexical unit indicating the end of the response is predicted. Classifier 965 and generation mechanism 970 can be... Figure 9B The classifier 950 and generation mechanism 955 operate similarly. Generation mechanism 970 selects or samples each sequentially output lexical based on the corresponding predicted probability and appends it to the output of the previous pass, thereby sequentially generating each lexical until a symbol or lexical indicating the end of the response is selected or sampled. These and other architectures described herein are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0116] Other variations are also within the spirit of this disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments are shown in the accompanying drawings and have been described in detail above. However, it should be understood that this disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined by the appended claims.

[0117] In the context of describing the disclosed embodiments (especially in the context of the following claims), the terms “a” and “an”, as well as “the” and similar designations, shall be interpreted to cover both the singular and plural, unless otherwise indicated herein or obviously contradicted by the context, and are not defined as terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” shall be interpreted as open-ended terms (meaning “including, but not limited to”). When unmodified and referring to a physical connection, “connection” shall be interpreted as partially or completely contained within, attached to, or joined together, even with intervening elements. Unless otherwise indicated herein, the enumeration of ranges of values ​​herein is intended only as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually enumerated herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” shall be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise stated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean a suitable subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0118] Unless otherwise specifically stated or otherwise obviously contradicted by the context, connective language (such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C") is also understood, along with the context in which it is generally used, to mean that an item, term, etc., can be any non-empty subset of the set A or B or C, or A and B and C. For example, in an exemplary example of a set with three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the existence of at least one of A, at least one of B, and at least one of C. Furthermore, unless otherwise stated or contradicted by the context, the term "multiple" means a plurality of states (e.g., "multiple items" means multiple items). In at least one embodiment, the number of items in the plurality is at least two, but may be more when explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly indicated from the context, the phrase “based on” means “at least partially based on” or “at least based on”, not “based on only”.

[0119] The operations of the processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise obviously contradicted by the context. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions, and are executed jointly on one or more processors via hardware or a combination thereof in the form of code (e.g., executable instructions, one or more computer programs, or one or more application programs). In at least one embodiment, the code is stored on a computer-readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include transient signals (e.g., transmission of transient electrical or electromagnetic signals), but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a collection of one or more non-transitory computer-readable storage media on which executable instructions (or other memory for storing executable instructions) are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the collection of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all code, while the multiple non-transitory computer-readable storage media collectively store all code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, the non-transitory computer-readable storage media stores the instructions, and the main central processing unit (“CPU”) executes some instructions, while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of instructions.

[0120] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that individually or jointly perform the operations of the processes described herein, and such a computer system is configured with available hardware and / or software to perform the operations. Further, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment, it is a distributed computer system comprising multiple devices operating differently, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0121] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and is not intended to limit the scope of this disclosure unless otherwise required. The language in the specification should not be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0122] All references cited in this article (including publications, patent applications, and patents) are incorporated into this article by reference to the same extent that each reference is individually and specifically indicated to be incorporated into this article by reference and is listed in its full text.

[0123] In the specification and claims, the terms “coupled” and “connected” and their derivatives may be used. It should be understood that these terms are not intended to be synonyms with each other. Rather, in specific examples, “connected” or “coupled” can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0124] Unless otherwise explicitly stated, in some embodiments it should be understood that throughout this specification, terms such as “processing,” “operation,” “calculation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device for manipulating and / or converting data represented in physical quantities (such as in electronic form) in the registers and / or memory of the computing system into other data similarly represented in physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0125] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and agents. Moreover, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods and a method can be considered a system.

[0126] This document may refer to obtaining, acquiring, receiving, or inputting analog or digital data to a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be implemented by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, the provisioning, outputting, transmitting, sending, or presenting analog or digital data may also be mentioned. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0127] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although a particular distribution of responsibilities may be defined above for descriptive purposes, individual functions and responsibilities may be distributed and divided in different ways depending on the environment.

[0128] Furthermore, although the subject matter has been described using language specific to structural features and / or methodological actions, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A system comprising a network interface controller (NIC), the network interface controller being configured to: Generate a first network packet with first metadata, wherein, The first metadata indicates a first association with a plurality of parallel processing units used to perform distributed artificial intelligence (AI) operations; After generating the first network data packet, a second network data packet with second metadata is generated, wherein the second metadata indicates a second association with the plurality of parallel processing units for performing the distributed AI operation; Based on a comparison between the second metadata and the first metadata, it is determined that the second network data packet has a higher priority than the first network data packet; and The second network data packet is provided to the network device before the first network data packet is provided.

2. The system as claimed in claim 1, wherein, The first network packet containing the first metadata is generated for an AI kernel executed using the first parallel processing unit among the plurality of parallel processing units.

3. The system as described in claim 1, wherein, The first association includes one or more target parallel processing units among the plurality of parallel processing units.

4. The system as claimed in claim 1, wherein, The first metadata includes at least one of the following: First communication set identifier; Data type; Collection data volume indicator; or Operation type.

5. The system as described in claim 4, wherein, The second metadata includes a second communication set identifier, which is different from the first communication set identifier.

6. The system as claimed in claim 1, wherein, The network device is a network switch configured to process network traffic based on AI-related metadata.

7. The system as claimed in claim 1, wherein, The distributed AI operation includes one or more of the following: Broadcast operation; Full reduction operation; Reduction operation; Full collection operation; Collection operation; Distributed operations; Reduce-distributed operation; or Barrier operation.

8. A network switch, the network switch comprising: Memory; as well as A processing unit, coupled to the memory, is shown that the processing unit is used for: Receive the first network data packet; After receiving the first network data packet, the second network data packet is received; Based on the first metadata of the first network data packet, a first rule for processing the first network data packet is determined, and based on the second metadata of the second network data packet, a second rule for processing the second network data packet is determined, wherein the first metadata indicates a first association with a plurality of parallel processing units for performing distributed artificial intelligence (AI) operations, and the second metadata indicates a second association with the plurality of parallel processing units for performing the distributed AI operations; Based on a comparison between the second metadata and the first metadata, it is determined that the second network data packet has a higher priority than the first network data packet; and The second network data packet is provided to the first network device before the first network data packet is provided.

9. The network switch as described in claim 8, wherein, In order to determine the first rule for processing the first network data packet and the second rule for processing the second network data packet, the processing unit is configured to receive one or more control messages, the one or more control messages including the first rule for processing the first network data packet and the second rule for processing the second network data packet.

10. The network switch as described in claim 9, wherein, The one or more control messages are generated for an artificial intelligence (AI) kernel executed using the first parallel processing unit among the plurality of parallel processing units.

11. The network switch as described in claim 8, wherein, In order to determine the first rule for processing the first network packet and the second rule for processing the second network packet, the processing unit is configured to: Send a query to the second networked device; and In response to the query, a reply is received from the second network device, the reply including the first rule for processing the first network data packet and the second rule for processing the second network data packet.

12. The network switch as described in claim 11, wherein, The response is generated for an artificial intelligence (AI) kernel executed by the first parallel processing unit among the plurality of parallel processing units.

13. The network switch as described in claim 8, wherein, The first metadata includes at least one of the following: First communication set identifier; Data type; Collection data volume indicator; or Operation type.

14. The network switch as described in claim 13, wherein, The first metadata includes a first set data volume indicator, which indicates that the number of network packets generated by the first communication set is less than a first data volume threshold, and wherein the processing unit is further configured to store the first network packets in a buffer.

15. The network switch as described in claim 8, wherein, The first rule for processing the first network packet, determined based on the first metadata associated with the first network packet, includes at least one of the following: Priority scheduling rules for the first network data packet; Buffering decision for the first network data packet; or Network routing rules for the first network data packet.

16. A method, the method comprising: Receive the first network data packet; After receiving the first network data packet, the second network data packet is received; A first rule for processing the first network data packet is determined based on the first metadata of the first network data packet, and a second rule for processing the second network data packet is determined based on the second metadata of the second network data packet, wherein the first metadata indicates a first association with a plurality of parallel processing units for performing distributed artificial intelligence (AI) operations, and the second metadata indicates a second association with the plurality of parallel processing units for performing the distributed AI operations; Based on a comparison between the second metadata and the first metadata, it is determined that the second network data packet has a higher priority than the first network data packet; and The second network data packet is provided to the first network device before the first network data packet is provided.

17. The method of claim 16, wherein, Determining the first rule for processing the first network data packet and the second rule for processing the second network data packet includes: receiving one or more control messages, the one or more control messages including the first rule for processing the first network data packet and the second rule for processing the second network data packet.

18. The method of claim 17, wherein, The one or more control messages are generated for an artificial intelligence (AI) kernel executed using the first parallel processing unit among the plurality of parallel processing units.

19. The method of claim 16, wherein, Determining the first rule for processing the first network data packet and the second rule for processing the second network data packet includes: Send a query to the second networked device; and In response to the query, a reply is received from the second network device, the reply including the first rule for processing the first network data packet and the second rule for processing the second network data packet.

20. The method of claim 19, wherein, The response is generated for an artificial intelligence (AI) kernel executed by the first parallel processing unit among the plurality of parallel processing units.

Images