Orchestration of distributed reasoning operations

Abstract

The invention relates to orchestration of distributed reasoning operations. The approaches presented herein provide for management of resources to be used to process requests, such as possibly involving orchestration of nodes for inference requests. Upon receiving an inference request, an orchestrator may determine a sequence of context nodes and inference nodes to be used to process the inference request based in part on the determined category of inference to be performed. The orchestrator may append metadata identifying the sequence to the inference request, and may transmit the appended request in order to one or more first nodes. If the nodes have network programmable devices or similar capabilities, the requests may be forwarded to the nodes in order without having to return back to orchestrators between the nodes.

Description

Technical Field

[0001] This disclosure relates to the management of sets of computing resources, and in at least one embodiment relates to accelerating the deployment and orchestration of distributed inference devices by using one or more programmable network devices. Background Technology

[0002] In various computing environments (specifically, computing environments using large language models (LLMs) or other models with a large number of parameters), it can be beneficial to distribute sets of computational operations across large sets of computing resources, such as physical or virtual servers. Even when distributed across large resource sets, tasks such as training and performing inference can be time-consuming and resource-intensive. However, in many instances, it is difficult to meet this higher demand while maintaining various performance or quality-of-service metrics. Several frameworks have been proposed for building and optimizing LLM applications, which run on CPU-based servers and provide development kits or interfaces that allow users to utilize the different entities required for LLM-based applications, such as various inference systems, models, and vector databases (VDBs). These frameworks primarily focus on building a single LLM pipeline for a single application and anticipate users setting up and configuring various entities individually, while more complex LLM-based applications require more dynamic pipelines. This may occur, for example, when inference requests require different or multiple models along with specific contextual information. Various artificial intelligence (AI) service providers have begun offering different categories of services and operational models, requiring more complex orchestration of inference requests. For example, inference requests can be directed to a model based on user category and / or user requirements (e.g., accuracy level and / or task type). Furthermore, orchestration may become more complex and critical as it may require the use of proxies or a hybrid of proxies to serve requests sequentially or in parallel across multiple dedicated models. Current approaches do not provide a mechanism for efficiently and dynamically executing inference requests based on the configurations of various service providers without using unnecessary hardware resources (such as multiple CPU cores). Moreover, existing models are often simple and monolithic, failing to support variable-length chains. Attached Figure Description

[0003] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0004] Figure 1 The illustration shows an example network architecture that can be used to perform computational operations according to at least one embodiment;

[0005] Figure 2 The diagram illustrates the action flow to be performed during the initialization phase according to at least one embodiment;

[0006] Figure 3A The diagram illustrates a flow of actions to be performed during the operation phase according to at least one embodiment;

[0007] Figure 3B The diagram illustrates an action flow to be performed during the operation phase using metadata attached to the request, according to at least one embodiment;

[0008] Figure 4A The illustration shows an example process that can be performed on a resource set during the initialization phase according to at least one embodiment;

[0009] Figure 4B The illustration shows an example process that can be performed on a resource set during the operation phase according to at least one embodiment;

[0010] Figure 4C The illustration shows an example process that can be executed according to at least one embodiment to process inference requests using a resource set;

[0011] Figure 4D The illustration shows an example process that can be performed on a resource set during the initialization phase according to at least one embodiment;

[0012] Figure 5 An example data center system according to at least one embodiment is illustrated;

[0013] Figure 6 It is a block diagram illustrating a computer system according to at least one embodiment;

[0014] Figure 7 This is a block diagram illustrating a computer system according to at least one embodiment;

[0015] Figure 8 The illustration shows a computer system according to at least one embodiment;

[0016] Figure 9 The illustration shows a computer system according to at least one embodiment;

[0017] Figure 10 The illustration shows an exemplary integrated circuit and a related graphics processor according to at least one embodiment;

[0018] Figure 11A , Figure 11B The illustration shows an exemplary integrated circuit and a related graphics processor according to at least one embodiment;

[0019] Figure 12 The illustration shows a computer system according to at least one embodiment;

[0020] Figure 13A The illustration depicts a parallel processor according to at least one embodiment;

[0021] Figure 13B The illustration shows a partitioning unit according to at least one embodiment; and

[0022] Figure 14 The illustration shows at least a portion of a graphics processor according to one or more embodiments. Detailed Implementation

[0023] Various embodiments are described in the following description. Specific configurations and details are set forth for illustrative purposes in order to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments can be practiced without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0024] The approaches described in various illustrative embodiments can provide management of computing resources for various computing tasks. This can include accelerating the deployment and / or orchestration of distributed inference devices by using one or more programmable network devices, such as one or more data processing units (DPUs) or network interface cards (e.g., SmartNICs). To avoid the overhead of existing solutions where the central processing unit (CPU) is directly involved in all operations and support for complex pipelines is limited, programmable network devices can be used to perform tasks such as orchestration, such that requests are directed to one or more nodes capable of performing inference operations on one or more tasks. This can include sending individual requests to one or more context nodes to obtain information useful for inference, and sending requests to one or more inference nodes (or computing / processing devices capable of performing at least one inference operation) to actually perform inference. Dynamic pipelines can be selected for specific inference operations, and the pipelines can be configured according to the selected... The pipeline (of nodes to be used in the operation) appends the ordering and selection of nodes to the request as metadata. Therefore, this pipeline can be at least partially request-aware or request-dependent, and can be adapted as appropriate. The format of the metadata allows these different nodes to communicate without sending requests and / or responses back to the orchestrator after each step. Various types of context can be obtained in parallel or sequentially from one or more context nodes, and the type of inference node to be used can be based in part on factors such as user preferences and the category associated with the received request. Different inference nodes can contain different AI models, and in some cases, multiple inferences can be generated and compared to select the final output. This approach can provide a future-proof network acceleration solution to address the above and other such challenges and requirements, such as those of future LLM-based applications.

[0025] In view of the teachings and suggestions contained herein, it will be apparent to those skilled in the art that variations of this functionality and other such functionalities may also be used within the scope of various embodiments.

[0026] Figure 1 An example system 100, according to at least one embodiment, is illustrated and can be used to perform computational operations (such as those related to distributed inference requests). As mentioned, various computing environments exist in which data processing can be performed. Examples of such environments include data center and multi-tenant resource provider environments, such as those providing cloud computing service sets. These environments can be used to perform various computational operations on behalf of several different users, parties, or entities (e.g., by using pools of available resource capacity). Figure 1 The illustration depicts an example architecture 100 for such an environment that can be used according to at least one embodiment. In this example, a user can use client device 102 to submit one or more requests to access one or more resources, or to use one or more resources to perform a task, and other such options. Such requests can be submitted via at least one network 104 (such as the Internet or a cellular network) and received by an interface, address, or endpoint in a shared resource environment 106. The request can be received by an interface, such as an application programming interface (API) of interface layer 108. In this example, it may first be necessary to analyze the request from client device 102 to determine whether the client device, the user, or other entities associated with the request can access one or more resources to be used to process the request; and to determine whether the permitted access type allows the execution of the requested operation.

[0027] In this example, the requested information may be directed to Access Control Manager 112 or other such components, systems, or services. Access Control Manager 112 may perform various tasks to determine and / or manage access to a set of shared resources, such as extracting relevant information from a received request and comparing the requested information with information in Account Store 116 or other such locations. This operation, which may be performed by or in conjunction with Account Manager 120, may be used to determine whether the request is associated with a valid account (such as an account maintained by a user at the provider of Shared Resource Environment 106) associated with the Shared Resource Environment. Once determined, this account information can be used to determine the type of access permitted to perform one or more operations associated with the request. This may include, for example, determining (or verifying) an authorized user identifier associated with the request and then using that user identifier to determine the access permissions associated with that user identifier, such as access permissions that may be stored in Access Control Data Store 118 or other such locations. In at least one embodiment, the access control manager 112 may include various modules for performing specific tasks, such as authorization and authentication modules, or may run on a network server that also has these modules available for use with the access control manager 112, as well as other such options.

[0028] Once a set of access permissions associated with a request is identified, the access control manager 112 (or an associated process) can determine whether the necessary permissions exist in that set to process the request received from the client device and associated with a user identifier. If the appropriate permissions are determined to exist or be available, the access control manager 112 can direct the request information to one or more shared resources (and / or potentially private resources) within the shared resource environment 106. In some embodiments, the access control manager 112 may work in conjunction with the resource manager 110 to determine a specific instance of the resource type to be used to perform the operation related to the request, wherein the resource manager 100 may perform other types of operations as needed, such as allocating additional capacity for a resource, starting a new computing instance, or performing another such task associated with the request.

[0029] When allocating resources, resource manager 110 may allocate them at least in part based on the capabilities (and availability and other such factors) of different types of resources. This may include, for example, resources 122 capable of performing various tasks, such as physical servers, virtual servers, or compute instances. There may also be various types of resources 124 containing specific devices or components, or otherwise possessing additional or specific capabilities that enable these resources 124 to perform different types of tasks. For example, in this example, there may be resources 124, each including one or more programmable network devices (PNDs) 126 that can perform additional tasks, such as parsing, reformatting, and transmitting requests to perform one or more compute operations. Resource manager 110 may then allocate one or more of these types of resources 122, 124 in part based on the various types of operations a given user might require. An example compute device 130 corresponding to one type of resource having PNDs is illustrated. In this example, in addition to conventional components such as CPU 132, one or more graphics processing units (GPUs) 142, at least one DIMM 134 or other memory devices, and PCI devices 136, the computing device 130 also includes, for example, a pair of programmable network devices (PNDs) in the form of a NIC 138 and at least one DPU 140. As mentioned, these PNDs can be used to perform specific tasks that might be impossible using a type of resource 122 that does not include such devices or functions.

[0030] In at least one embodiment, resource manager 110 may work in conjunction with orchestrator 114, which is capable of directing requests to resources required to perform a given request. For example, an inference request may need to be forwarded (or a separate request may be transmitted) to a resource acting as a context node to provide context information (or other such data) determined to be relevant to the request or an operation to be performed in response to the request. An inference request may also need to be forwarded to at least one resource acting as an inference node to perform at least one inference operation on the request, which may include using context information received from at least one context node.

[0031] To simplify tasks such as orchestrating distributed inference requests, approaches according to various embodiments can leverage one or more network acceleration solutions. This can include, for example, using centralized and distributed orchestrators 114 to efficiently configure relevant components across a relevant computing environment. Network acceleration approaches can help minimize communication overhead, thereby allowing direct communication between different entities involved in inference operations. Such approaches can also allow for more efficient use of hardware resources and reduced power consumption, such as freeing up CPU-based servers to perform more "critical" tasks where programmable network devices can be used to perform orchestration. These approaches can also support advanced LLM applications that can use different categories of services and multiple models.

[0032] When used with inference requests or tasks, components or services such as orchestrator 114, which may be referred to as "inference orchestrators," can help efficiently deploy and orchestrate distributed inference-related operations and resources. Orchestrator 114 can handle any and / or all operations related to the initialization and operation phases, which may involve tasks such as configuring different entities based on workload and user configuration, and the operation phase may involve tasks such as receiving requests and scheduling / distributing requests to appropriate entities. As an example, during the operation phase, orchestrator 114 may receive an inference request from client device 102 (such as a desktop computer or operator terminal) and attempt to classify the request. Orchestrator 114 can then obtain the appropriate context for the request from the appropriate context node in the network and enable the inference request to be executed on the relevant inference node. Context data can be in any suitable form, such as text, audio, video, images, code, potential vectors, etc. After completing this execution, orchestrator 114 can return an inference response to client device 102 and / or forward it to a target destination or address. The orchestrator 114 can also attach additional metadata to the request so that different entities can communicate directly, thereby eliminating unnecessary communication overhead.

[0033] Figure 2The illustration shows an overview 200 of a task flow to be performed by inference orchestrator 204 during an example initialization phase according to at least one embodiment. In this example, a user or administrator may use client device 208 to provide information about the configuration required to deploy and orchestrate inference operations. Inference orchestrator 204 may process this information for application of appropriate configuration. In this example, the received configuration information may include descriptions of various nodes or node types to be used. This description may include details, for example, of the physical machines and / or virtual machines to be used to deploy the various nodes, such as inference node 206, context node 202, and model repository 210, as well as other such options. For example, there may be processing nodes (such as individual processing units, cores, or other such physical or virtual elements with processing capabilities) indicated in the inference pipeline that perform tasks potentially unrelated to inference, such as upscaling or super-resolution of AI-generated images. Users may provide relevant information for deploying nodes as a node cluster to work together to run one or more containerized applications, such as in the form of a Kubernetes cluster. This information can also specify whether certain nodes should be equipped with network accelerators (e.g., DPUs and smart NICs) and whether network acceleration should be supported.

[0034] The inference orchestrator 204 can also specify the request format for these inference operations, such as potentially involving HTTP / REST or L4+ protocol headers, to ensure that the inference orchestrator 104 can properly parse request messages and / or packets. One or more users can provide metrics to be used to orchestrate inference requests. In one example, a user or administrator can specify different models for different types of requests. Request types can be defined based on request size (e.g., hint length), request category (e.g., free or premium user), request requirements (e.g., minimum required accuracy that may affect the size of the selected model), and / or request task type (e.g., encoding hints, translation hints, and / or image generation hints), and other such options. Additionally, users can specify how much hardware resources (e.g., how many GPUs) or how much resource capacity should be used for each type of request. The inference orchestrator 204 can specify one or more inference modes, such as whether various inference operations should be performed in a centralized or distributed manner. The inference orchestrator 204 can also specify one or more supporting models and the categories of requests for which each supporting model should be used. The inference orchestrator 204 can also specify one or more policies, such as those related to scheduling, quality of service, or other similar aspects. This policy information can be used to schedule various inference requests and / or prioritize them, as determined based on the request types discussed earlier. Additionally, scheduling and prioritization may also affect how certain requests are traversed across the network based on various load balancing, routing, and other such decisions.

[0035] The inference orchestrator 204 can use this configuration information, along with other relevant configuration information, to deploy various entities. This may involve interacting with at least one entity (such as inference node 206) and other objects or components (such as one or more context nodes, model repositories, and / or attached inference nodes). In this example, model repository 210 can be accessed by the inference orchestrator 204 to load appropriate models and model weights on the relevant inference nodes. For example, this could include loading model weights on a graphics processing unit (GPU) or AI accelerator. The inference orchestrator 204 can also prepare and configure nodes equipped with programmable network devices (e.g., data processing units (DPUs)) to enable these devices to parse metadata attached to requests and redirect or forward responses (and new requests) to the appropriate nodes. After performing this configuration and deployment, the underlying inference infrastructure can be ready to serve specific types of AI requests.

[0036] After successful configuration and deployment, in at least one embodiment, the inference system 300 (such as...) Figure 3AThe illustrated inference system can enter the operational phase. In this phase, the inference orchestrator 304 can perform various tasks to serve, for example, received inference requests (or other requests requiring inference or other types of processing). Inference requests can be transmitted from client 302 and can be directed to inference orchestrator 304. As previously discussed, inference orchestrator 304 can parse the received request according to a previously defined format. Inference orchestrator 304 can attempt to classify the request in part based on this (and potentially other relevant) information, and then select one or more appropriate context nodes and one or more inference nodes. For example, classification can include analyzing prompts included in the request to attempt to determine a type of inference operation to be performed on the prompts. One or more context nodes 306 can be selected or identified, which can provide contextual information useful for one or more corresponding inference operations, and this contextual information can be related to, for example, chat history or additional information from a vector database (VDB). One or more inference nodes 308 can also be selected to perform one or more inference operations on behalf of a request, such as corresponding to a computation node that uses or hosts an AI model determined to satisfy one or more requirements of the request. The inference orchestrator 304 can access multiple inference nodes 308, which may have different accuracy levels, sizes, and / or expertise, and other such differentiating aspects. For example, if multiple models with different expertise are available, such as each model being specialized for a particular task, the inference orchestrator 304 can use vector embedding-based techniques, for example, to select an appropriate model or perform inference on all models. As discussed, other types of nodes can also be used, such as one or more processing nodes 310 used to perform processing that may be unrelated to inference and can provide other types of information to be transmitted to the inference orchestrator 304.

[0037] After selecting one or more appropriate nodes, the inference orchestrator 304 can attach additional metadata to the original request to facilitate direct communication between various nodes or other such entities, at least when the inference nodes are configured in a "distributed" or similar operating mode. Figure 3A In the centralized inference model illustrated, the inference orchestrator 304 can remain in a loop without adding any metadata to the request. (As shown in...) Figure 3BAs illustrated in operating mode 350, this system can be configured to operate in a distributed mode, where requests do not need to be forwarded back to the inference orchestrator 304 after each individual node in the inference pipeline, provided that at least the relevant node has the capability to parse, reformat, and / or transmit the modified request. In distributed operating mode, the inference orchestrator can add or append metadata to the received requests. The added metadata may include information such as the address and the request of one or more subsequent nodes in the inference pipeline for which a type of inference is to be performed. The added metadata may also include information about the expected recipient of the final result and other such information. If one or more operations are to be performed in parallel, such as performing an inference operation using two different models that will produce two different results, the metadata may also specify criteria for selecting one of the results or for combining the results together into a single response, and other such options. For example, the metadata may specify a formula for generating the result score and then indicate whether to select the result with the highest score, etc. Embedding this information in the metadata avoids the need to return to the inference orchestrator for computation, result selection, or other such operations unless these operations cannot be performed on one or more relevant nodes.

[0038] When metadata is added or appended, various nodes can utilize programmable network devices 352, 354 (e.g., DPUs) that are associated with or in conjunction with a given node (such as context node 306 or inference node 308) to facilitate the integration of metadata processing and / or parsing within the node without modifying existing software. In one example, the DPU in a node can receive a request with added metadata and can provide the software with the request in an appropriate format. The DPU can then reformat the request using the metadata as appropriate and send the request to the next node. The inference orchestrator 304, together with the programmable network device 354, provides acceleration for inference operations. If one or more nodes are not equipped with a programmable network device (e.g., a DPU), the inference orchestrator 304 can add itself as one of the nodes in the metadata and can then participate only in a subset of inter-node communication where a particular node may not have the appropriate capability to process and forward requests without sending the requests back to the inference orchestrator 304.

[0039] Inference orchestrator 304 can send received inference requests (including attached metadata) to one or more relevant nodes, including context node 306 and / or inference node 308. In centralized mode, this process can be executed sequentially until inference orchestrator 304 receives responses from all relevant nodes, at which point inference orchestrator 304 can generate a final response to the request. Inference orchestrator 304 can then send the response to client 302 and / or another designated recipient. For example, in distributed inference mode, the final response can also be sent directly from the last inference node, effectively bypassing inference orchestrator 304. If inference is performed on multiple models, inference orchestrator 304 can always be involved in the response process. Inference orchestrator 304 can use various methods to generate the final response or combine responses. For example, in cases where responses from different models are complementary, a node can combine all responses. This node can be a programmable network device (e.g., a DPU), such as a programmable network device running inference orchestrator 304 or a programmable network device from inference node 308. Because multi-model inference can be performed in parallel, information can be added via metadata to enable all inference nodes to aggregate their responses in one place (such as a specific inference node). In various situations, it may be desirable to select only one response. For example, the vector embeddings of all inference responses can be computed, and then the result that most closely resembles the requested vector embedding can be identified and returned. In such cases, the inference orchestrator 304 can perform and / or coordinate additional steps or tasks as needed before returning a response to the client 302 or other designated recipient.

[0040] This approach offers several advantages over existing solutions. For example, the inference orchestrator can concurrently retrieve context from multiple context nodes. This could include information such as chat history and other data from vector databases. The inference orchestrator can select one or more appropriate inference nodes, partly based on factors such as user preferences and request type and / or category. For instance, a prompt received in a request might require an answer with a description and an image, where determining the answer, generating a simple language description, and generating an appropriate image might each require different machine learning models, which might be executing on different compute nodes. The inference orchestrator can also enable inference to be performed on multiple AI models (e.g., by using a set of inference microservices, such as those provided by NIM or AI Optimization Servers from NVIDIA), including specialized models such as proxy models, multimodal models, and mixtures of models with different sizes and / or accuracies, among other such options. The inference orchestrator can add metadata to the received requests, which can be used to mitigate communication overhead in inference operations, such as when a network-accelerated node can parse the metadata and redirect the request to the next node. In at least one embodiment, where the orchestrator sends requests to multiple inference nodes, the inference orchestrator may also combine, merge, and / or select one or more specific responses from the multiple inference nodes.

[0041] This type of approach can provide a framework for requests to be processed differently based on various factors that can be associated with them. For example, different requests may have different Quality of Service (QoS) requirements, which may necessitate directing these requests (or information about these requests) to different sets or multiple sets of nodes. This type of approach can also help minimize communication overhead by allowing different paths and communication types, where nodes can communicate directly with each other without having to go through a central orchestrator on every communication. While a central inference orchestrator can be used, the orchestrator does not need to be the intermediary for all communication. The inference orchestrator can analyze a request upon receiving it and make various decisions about the required information and / or the tasks to be performed on the request. The orchestrator can then attach appropriate metadata to the request, partly based on these decisions, whereby the metadata may include information such as which nodes should participate in the corresponding operation. The orchestrator can then send the request with the attached metadata to the first node (or first set of nodes) selected to process the request. Nodes may be equipped with programmable network devices or other such mechanisms or capabilities that can parse the metadata and extract the information needed for that particular node to perform the operation. After performing one or more corresponding operations, a node can form or generate a new response that is constructed based on the previous request and metadata to include one or more results of the one or more operations. This new response can then be forwarded to the next node (or set of nodes) in a predetermined order, rather than being returned to the orchestrator, unless this is the final response intended to be returned to the orchestrator. If a given node does not have a programmable network device or similar capability, the request to that node can be directed back to the orchestrator for updating and, as appropriate, sending to the next node.

[0042] Figure 4AAn example process 400, according to at least one embodiment, can be executed during the initialization phase of a resource set. It should be understood that, for this process and other processes disclosed herein, additional, fewer, or alternative steps may be performed in a similar or alternative order or at least partially in parallel within the scope of various embodiments. Furthermore, although this example process discusses inference orchestrators, contexts, and inference nodes, additional or alternative orchestration devices or processes, as well as additional or alternative types of nodes, may also be used unless specifically stated otherwise. In this example, information 402 required for the deployment and orchestration of a category of inference operations is received from a user, operator, or other authorized entity. This information may relate to various types of required nodes, the configuration of these nodes, dependencies between these nodes, the capabilities of these nodes, etc. Such information for each of several different categories may be received from one or more different entities. In this example, an inference orchestrator 404 may be used to process this information, and in at least one embodiment, the inference orchestrator may utilize at least one programmable network device. The inference orchestrator can perform several different tasks as part of the initialization phase, and although shown sequentially here, these tasks can be performed in different orders or at least partially in parallel by one or more orchestrators or devices communicating with the orchestrators, and other such options. In addition to the tasks illustrated but within the scope of at least one embodiment, many additional tasks can be performed. A certain number and one or more types of nodes (e.g., inference nodes, context nodes, or processing nodes) to be used for operations of this category can be determined at 406, including any required capabilities of these nodes (e.g., servers with NPD installed). The request format to be used by all nodes and the inference orchestrator at 408 for parsing, updating, or reformatting requests can be determined at 410. The inference mode, such as whether centralized or distributed inference is used for inference of a given category, can be determined at 410. For example, if an inference request only requires processing by one node, the small overhead of distributed processing can be avoided by selecting centralized processing, which does not involve adding metadata. Since a given inference request may require multiple inference operations using different models, the model to be used for each inference operation in this category can be determined at 412. One or more applicable strategies for the category can be determined, such as those relating to how to handle conflicts, errors, or competition outcomes. The node sequence for the category can be determined based on one or more strategies. The nodes for the category can be configured to perform specific types of operations, such as including, as appropriate, loading one or more machine learning models. Once all relevant determinations are made, the inference orchestrator can provide, on the other hand, an optional pipeline for the category and the inference operations for each category, which the inference orchestrator can select and use in response to an inference request for the appropriate inference category.

[0043] Figure 4B An example process 420, which can be implemented to perform orchestration on a received inference request according to at least one embodiment, is illustrated. In this example, a inference request is received 422 to an inference orchestrator. The inference orchestrator can analyze 424 (or cause analysis) the inference request to determine the category of inference to be performed. Then, 426 the inference pipeline can be selected, such as in... Figure 4A The process generates one pipeline corresponding to the determined category. A sequence of 428 nodes can be determined based on the selected pipeline, where these nodes may have specific types and capabilities to be used to process inference requests. The inference orchestrator can attach 430 metadata to the inference request, which includes at least information about the node sequence, including information identifying the nodes. A request with attached metadata can be sent to one or more nodes according to the sequence specified in the metadata. For example, a request can be sent to a first context node or to multiple context nodes in parallel to obtain context information to be used to perform inference on the request. The programmable network device on the receiving node or node can parse the request and provide the request information in an appropriate format to the relevant software executing on one or more corresponding nodes. After processing has been completed, such as when the context node has identified the context information or the inference node has generated an inference result, the programmable network device can be allowed to reformat the request as appropriate, and the request can be updated using result information from one or more relevant nodes. It can be determined whether there are more nodes in the sequence, and if so, the process can continue by sending the current request to one or more next nodes in the sequence. As disclosed, in distributed operation mode and with the relevant nodes possessing PND or similar capabilities, the request can be sent without returning to the inference orchestrator. Once it is determined that no more nodes exist in the 438 sequence, a final response (439) can be generated and sent in response to the inference request. This response can be generated by the inference orchestrator or a node with the capability, and can be sent back to the source of the request or another appropriate recipient, among other such options.

[0044] Figure 4C An example process 440, according to at least one embodiment, can be performed on a received request using an inference orchestrator and a node with programmable network devices or similar functionality. This process illustrates falling into... Figure 4BAn example within the general orchestration process 420. In this example, a request is received from the client 442. Then, one or more context nodes 444 can be sent to one or more context nodes determined for that request. Nodes can identify and acquire relevant context through any appropriate positioning mechanism, such as a vector-based search of a vector database. Context information 446 can be received from these context nodes, and then the acquired context information 448 can be appended to the received request (via the inference orchestrator or one of the PNDs). One or more appropriate inference nodes can also be selected 450 for the request. An inference request 452 can then be sent to the selected inference nodes according to the determined sequence. An inference response 454 can then be combined via the inference orchestrator or one of the PNDs on one of the nodes. An inference response 456 can then be sent to the client.

[0045] Figure 4D The illustration depicts another example process 460, according to at least one embodiment, that can be performed on a received request using an inference orchestrator and a node with programmable network devices or similar functionality. This process illustrates falling into... Figure 4B Another example within the general orchestration process 420. In this example process 460, a request 462 is received from the client and directed to the inference orchestrator. 464 One or more appropriate context nodes and one or more inference nodes can be selected for the request. 466 Metadata specifying the node sequence (with node addresses) and other relevant information can be added to the request. 468 A new request 468 can be sent to the first context node of the sequence, and then 470 all components (e.g., subsequent nodes in the sequence) can be allowed to communicate directly without having to return to the inference orchestrator, unless one or more nodes in the sequence lack the appropriate capabilities. If more than one inference response exists, 472 (or selection) of inference responses can be combined, 472 (or selected), if needed and once an inference response can be sent back to the client (whether by the inference orchestrator or the final node of the sequence) as a final inference response, and other such options.

[0046] As mentioned, such approaches can help mitigate the communication overhead experienced in inference orchestration and can help avoid or reduce unnecessary use of processor (e.g., CPU) capacity experienced in existing CPU-based orchestration approaches. The embodiments disclosed herein can support dynamic and advanced inference pipelines, such as inference pipelines that can serve multiple request classes and support multi-node, multi-context distributed service systems. This solution can leverage programmable network devices or similar capabilities to perform orchestration and extend requests with specific metadata formats so that entities can communicate directly without sending requests / responses back to the orchestrator after each step. This solution can also support scheduling and / or selecting appropriate nodes for requests and dispatching them accordingly. As mentioned, request-dependent dynamic pipelines can also be used and can support different quality of service levels. This approach can also provide a network acceleration solution that can efficiently orchestrate inference requests in various distributed settings. In at least one embodiment, the inference orchestrator can encapsulate all the systems and methods required to deploy and orchestrate distributed inference operations. Such an inference orchestrator can be implemented, either entirely or partially, on a programmable network device (e.g., a DPU or NIC). An example implementation may use one or more NVIDIA DPUs and provide a DOCA service (e.g., a DOCA LangChain service) to perform inference orchestration on one or more DPUs.

[0047] Data Center

[0048] Figure 5 An exemplary data center 500 in which at least one embodiment can be used is shown. In at least one embodiment, the data center 500 includes a data center infrastructure layer 510, a framework layer 520, a software layer 530, and an application layer 540.

[0049] In at least one embodiment, such as Figure 5As shown, the data center infrastructure layer 510 may include a resource orchestrator 512, grouped computing resources 514, and node computing resources (“nodes CR”) 516(1)-516(N), where “N” represents a positive integer (which may be an integer “N” different from the integers used in other diagrams). In at least one embodiment, nodes CR 516(1)-516(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory storage devices 518(1)-518(N) (e.g., dynamic read-only memory, solid-state storage, or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR 516(1)-516(N) may be servers having one or more of the aforementioned computing resources.

[0050] In at least one embodiment, the grouped computing resources 514 may include individual groups of node CRs housed within one or more racks (not shown), or a plurality of racks housed within data centers (also not shown) in various geographical locations. In at least one embodiment, the individual groups of node CRs within the grouped computing resources 514 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0051] In at least one embodiment, resource orchestrator 512 may configure or otherwise control one or more nodes CR516(1)-516(N) and / or grouped computing resources 514. In at least one embodiment, resource orchestrator 512 may include a Software Design Infrastructure (“SDI”) management entity for data center 500. In at least one embodiment, resource orchestrator 512 may include hardware, software, or some combination thereof.

[0052] In at least one embodiment, such as Figure 5As shown, framework layer 520 includes a job scheduler 522, a configuration manager 524, a resource manager 526, and a distributed file system 528. In at least one embodiment, framework layer 520 may include a framework of software 532 supporting software layer 530 and / or one or more applications 542 supporting application layer 540. In at least one embodiment, software 532 or application 542 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 520 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 528 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 522 may include a Spark driver for facilitating the scheduling of workloads supported by various layers of the data center 500. In at least one embodiment, the configuration manager 524 may be able to configure different layers, such as the software layer 530 and the framework layer 520, which includes Spark and a distributed file system 528 for supporting large-scale data processing. In at least one embodiment, the resource manager 526 may be able to manage clustered or grouped computing resources mapped to or allocated to support the distributed file system 528 and the job scheduler 522. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 514 at the data center infrastructure layer 510. In at least one embodiment, the resource manager 526 may coordinate with the resource orchestrator 512 to manage these mapped or allocated computing resources.

[0053] In at least one embodiment, the software 532 included in the software layer 530 may include software used by at least portions of nodes CR516(1)-516(N), grouped computing resources 514, and / or the distributed file system 528 of the framework layer 520. In at least one embodiment, one or more types of software may include, but are not limited to, Internet web search software, email virus scanning software, database software, and streaming video content software.

[0054] In at least one embodiment, one or more applications 542 included in application layer 540 may include one or more types of applications used by at least portions of nodes CR516(1)-516(N), grouped computing resources 514, and / or the distributed file system 528 of framework layer 520. In at least one embodiment, one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, applications, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0055] In at least one embodiment, any of the configuration manager 524, resource manager 526, and resource orchestrator 512 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. In at least one embodiment, self-modification actions can mitigate potentially poor configuration decisions by data center operators of data center 500 and can prevent underutilization and / or poor performance of the data center.

[0056] In at least one embodiment, data center 500 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above with respect to data center 500. In at least one embodiment, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to data center 500 by using weight parameters calculated through one or more training techniques described herein.

[0057] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to utilize the aforementioned resources to perform training and / or inference. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0058] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 5It is used in systems for inference or prediction operations based at least in part on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0059] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0060] Computer System

[0061] Figure 6 This is a block diagram illustrating an exemplary computer system according to at least one embodiment. The exemplary computer system may be a system of interconnected devices and components, a system-on-a-chip (SoC), or some combination thereof formed with a processor, which may include an execution unit for executing instructions. In at least one embodiment, according to this disclosure, such as in the embodiments described herein, computer system 600 may include, but is not limited to, components such as processor 602 for employing execution units (including logic) to execute algorithms for process data. In at least one embodiment, computer system 600 may include a processor, such as those available from Intel Corporation of Santa Clara, California. Processor family, Xeon TM , Scale TM and / or StrongARM TM , Core TM or Nirvana TM The microprocessor can be used, although other systems (including PCs, engineering workstations, set-top boxes, etc.) with other microprocessors can also be used. In at least one embodiment, the computer system 600 can execute a version of the Windows operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (such as UNIX and Linux), embedded software, and / or graphical user interfaces can also be used.

[0062] The embodiments can be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol (IP) devices, digital cameras, personal digital assistants (“PDAs”), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor (“DSP”), a system-on-a-chip (SoC), a network computer (“Necks”), a set-top box, a network hub, a wide area network (“WAN”) switch, or any other system capable of executing one or more instructions according to at least one embodiment.

[0063] In at least one embodiment, the computer system 600 may include, but is not limited to, a processor 602, which may include, but is not limited to, one or more execution units 608 for performing machine learning model training and / or inference according to the techniques described herein. In at least one embodiment, the computer system 600 is a single-processor desktop or server system, but in another embodiment, the computer system 600 may be a multiprocessor system. In at least one embodiment, the processor 602 may include, but is not limited to, for example, a Complex Instruction Set Computer (“CISC”) microprocessor, a Reduced Instruction Set Computing (“RISC”) microprocessor, a Very Long Instruction Word (“VLIW”) microprocessor, a processor implementing instruction set combination, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 602 may be coupled to a processor bus 610, which allows data signaling between the processor 602 and other components in the computer system 600.

[0064] In at least one embodiment, processor 602 may include, but is not limited to, a Level 1 (“L1”) internal cache memory (“cache”) 604. In at least one embodiment, processor 602 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside externally to processor 602. Depending on specific implementation and requirements, other embodiments may also include a combination of internal and external caches. In at least one embodiment, register file 606 may store different types of data in various registers, including but not limited to integer registers, floating-point registers, status registers, and instruction pointer registers.

[0065] In at least one embodiment, an execution unit 608, including but not limited to logic for performing integer and floating-point operations, is also located in the processor 602. In at least one embodiment, the processor 602 may also include a microcode (“ucode”) read-only memory (“ROM”) storing the microcode of certain macro instructions. In at least one embodiment, the execution unit 608 may include logic for processing a packaged instruction set 609. In at least one embodiment, by including the packaged instruction set 609 in the instruction set of the general-purpose processor and the associated circuitry to be executed, operations used by numerous multimedia applications can be performed using packaged data in the processor 602. In at least one embodiment, numerous multimedia applications can be accelerated and executed more efficiently by performing operations on packaged data using the full width of the processor's data bus, eliminating the need to transfer smaller data units on the processor's data bus to perform one or more operations on one data element at a time.

[0066] In at least one embodiment, execution unit 608 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuitry. In at least one embodiment, computer system 600 may include, but is not limited to, memory 620. In at least one embodiment, memory 620 may be a dynamic random access memory (“DRAM”) device, a static random access memory (“SRAM”) device, a flash memory device, or other memory device. In at least one embodiment, memory 620 may store one or more instructions 619 and / or data 621 represented by data signals executable by processor 602.

[0067] In at least one embodiment, the system logic chip may be coupled to the processor bus 610 and the memory 620. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub (“MCH”) 616, and the processor 602 may communicate with the MCH 616 via the processor bus 610. In at least one embodiment, the MCH 616 may provide a high-bandwidth memory path 618 to the memory 620 for instruction and data storage, as well as for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 616 may direct data signals between the processor 602, the memory 620, and other components in the computer system 600, and bridge data signals between the processor bus 610, the memory 620, and the system I / O interface 622. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 616 may be coupled to the memory 620 via the high-bandwidth memory path 618, and the graphics / video card 612 may be coupled to the MCH 616 via an Accelerated Graphics Port (“AGP”) interconnect 614.

[0068] In at least one embodiment, the computer system 600 may use the system I / O interface 622 as a proprietary hub interface bus to couple the MCH 616 to the I / O controller hub (“ICH”) 630. In at least one embodiment, the ICH 630 may provide direct connectivity to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripheral devices to the memory 620, chipset, and processor 602. Examples may include, but are not limited to, an audio controller 629, a firmware hub (“Flash BIOS”) 628, a wireless transceiver 626, a data storage 624, a conventional I / O controller 623 including a user input and keyboard interface 625, a serial expansion port 627 (such as a Universal Serial Bus (“USB”) port), and a network controller 634. In at least one embodiment, the data storage 624 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0069] In at least one embodiment, Figure 6 A system including interconnected hardware devices or "chips" is shown, while in other embodiments, Figure 6 An exemplary SoC can be shown. In at least one embodiment, Figure 6 The devices shown can be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of the computer system 600 are interconnected using a Compute Fast Link (CXL) interconnect.

[0070] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 6 Used in systems for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases as described herein.

[0071] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0072] Figure 7 This is a block diagram illustrating an electronic device 700 for utilizing a processor 710 according to at least one embodiment. In at least one embodiment, the electronic device 700 may be, for example, but not limited to, a laptop computer, tower server, rack server, blade server, laptop computer, desktop computer, tablet computer, mobile device, telephone, embedded computer, or any other suitable electronic device.

[0073] In at least one embodiment, the electronic device 700 may include, but is not limited to, a processor 710 communicatively coupled to any suitable number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 710 is coupled using a bus or interface, such as I... 2 C-bus, System Management Bus (“Sambas”), Low Pin Count (LPC) bus, Serial Peripheral Interface (“SPI”), High Definition Audio (“HDA”) bus, Serial Advanced Technology Accessory (“SATA”) bus, Universal Serial Bus (“USB”) (versions 1, 2, 3, etc.) or Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, Figure 7 The system shown includes interconnected hardware devices or "chips," while in other embodiments, Figure 7 An exemplary SoC can be shown. In at least one embodiment, Figure 7 The devices shown can be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or some combination thereof. In at least one embodiment, Figure 7 One or more components are interconnected using Computational Fast Link (CXL) interconnects.

[0074] In at least one embodiment, Figure 7 This may include a display 724, a touchscreen 725, a touchpad 730, a near-field communication unit (“NFC”) 745, a sensor hub 740, a thermal sensor 746, a fast chipset (“EC”) 735, a trusted platform module (“TPM”) 738, a BIOS / firmware / flash memory (“BIOS, FW Flash”) 722, a DSP 760, a drive 720 (such as a solid-state drive (“SSD”) or a hard disk drive (“HDD”)), a wireless local area network unit (“WLAN”) 750, a Bluetooth unit 752, a wireless wide area network unit (“WWAN”) 756, a global positioning system (GPS) unit 755, a camera (“USB 3.0 camera”) 754 (such as a USB 3.0 camera), and / or a low-power double data rate (“LPDDR”) memory unit (“LPDDR3”) 715 implemented in, for example, the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0075] In at least one embodiment, other components may be communicatively coupled to processor 710 via the components described herein. In at least one embodiment, accelerometer 741, ambient light sensor (“ALS”) 742, compass 743, and gyroscope 744 may be communicatively coupled to sensor hub 740. In at least one embodiment, thermal sensor 739, fan 737, keyboard 736, and touchpad 730 may be communicatively coupled to EC 735. In at least one embodiment, speaker 763, earphone 764, and microphone (“mic”) 765 may be communicatively coupled to audio unit (“audio codec and Class D amplifier”) 762, which in turn may be communicatively coupled to DSP 760. In at least one embodiment, audio unit 762 may include, for example, but not limited to, audio encoder / decoder (“codec”) and Class D amplifier. In at least one embodiment, SIM card (“SIM”) 757 may be communicatively coupled to WWAN unit 756. In at least one embodiment, components such as WLAN unit 750, Bluetooth unit 752, and WWAN unit 756 may be implemented as next-generation form factors (“NGFF”).

[0076] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 7 Used in systems for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases as described herein.

[0077] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0078] Figure 8 A computer system 8100 according to at least one embodiment is shown. In at least one embodiment, the computer system 8100 is configured to implement the various processes and methods described throughout this disclosure.

[0079] In at least one embodiment, the computer system 800 includes, but is not limited to, at least one central processing unit (“CPU”) 802 connected to a communication bus 810 implemented using any suitable protocol, such as PCI (“Peripheral Component Interconnect”), Peripheral Component Interconnect Express (“PCI-Express”), AGP (“Accelerated Graphics Port”), HyperTransport, or any other bus or point-to-point communication protocol. In at least one embodiment, the computer system 800 includes, but is not limited to, main memory 804 and control logic (e.g., implemented in hardware, software, or a combination thereof), and data is stored in the main memory 804, which may take the form of random access memory (“RAM”). In at least one embodiment, a network interface subsystem (“network interface”) 822 provides an interface to other computing devices and networks for receiving data from and sending data to other systems using the computer system 800.

[0080] In at least one embodiment, the computer system 800 includes, but is not limited to, an input device 808, a parallel processing system 812, and a display device 806, which may be implemented using conventional cathode ray tube (“CRT”), liquid crystal display (“LCD”), light-emitting diode (“LED”) display, plasma display, or other suitable display technologies. In at least one embodiment, user input is received from the input device 808 (such as a keyboard, mouse, touchpad, microphone, etc.). In at least one embodiment, each module described herein may reside on a single semiconductor platform to form the processing system.

[0081] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 8 The system is used to perform inference or prediction operations based at least in part on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0082] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0083] Figure 9 A computer system 900 according to at least one embodiment is illustrated. In at least one embodiment, the computer system 900 includes, but is not limited to, a computer 910 and a USB flash drive 920. In at least one embodiment, the computer 910 may include, but is not limited to, any number and type of processors (not shown) and memory (not shown). In at least one embodiment, the computer 910 includes, but is not limited to, a server, a cloud instance, a laptop computer, and a desktop computer.

[0084] In at least one embodiment, the USB flash drive 920 includes, but is not limited to, a processing unit 930, a USB interface 940, and USB interface logic 950. In at least one embodiment, the processing unit 930 can be any instruction execution system, device, or apparatus capable of executing instructions. In at least one embodiment, the processing unit 930 can include, but is not limited to, any number and type of processing cores (not shown). In at least one embodiment, the processing unit 930 includes an application-specific integrated circuit (“ASIC”) optimized to perform any number and type of operations associated with machine learning. For example, in at least one embodiment, the processing unit 930 is a tensor processing unit (“TPC”) optimized to perform machine learning inference operations. In at least one embodiment, the processing unit 930 is a vision processing unit (“VPU”) optimized to perform machine vision and machine learning inference operations.

[0085] In at least one embodiment, the USB interface 940 can be any type of USB connector or USB receptacle. For example, in at least one embodiment, the USB interface 940 is a USB 3.0 Type-C receptacle for data and power. In at least one embodiment, the USB interface 940 is a USB 3.0 Type-A connector. In at least one embodiment, the USB interface logic 950 may include any amount and type of logic enabling the processing unit 930 to interface with a device (e.g., computer 910) via the USB connector 940.

[0086] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 9 Used in systems for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases as described herein.

[0087] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0088] Figure 10 Exemplary integrated circuits and associated graphics processors according to various embodiments described herein are illustrated, which can be manufactured using one or more IP cores. In addition to those illustrated, at least one embodiment may also include other logic and circuitry, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0089] Figure 10This is a block diagram illustrating an exemplary system-on-chip (SOC) integrated circuit 1000 manufactured using one or more IP cores according to at least one embodiment. In at least one embodiment, the SOC integrated circuit 1000 includes one or more application processors 1005 (e.g., CPUs), at least one graphics processor 1010, and may additionally include an image processor 1015 and / or a video processor 1020, any of which may be a modular IP core. In at least one embodiment, the SOC integrated circuit 1000 includes peripheral or bus logic, which includes a USB controller 1025, a UART controller 1030, an SPI / SDIO controller 1035, and an I... 2 S / I 2 C controller 1040. In at least one embodiment, the SOC integrated circuit 1000 may include a display device 1045 coupled to one or more of a High Definition Multimedia Interface (HDMI) controller 1050 and a Mobile Industrial Processor Interface (MIPI) display interface 1055. In at least one embodiment, storage may be provided by a flash memory subsystem 1060, which includes flash memory and a flash memory controller. In at least one embodiment, a memory interface may be provided via a memory controller 1065 for accessing an SDRAM or SRAM memory device. In at least one embodiment, some integrated circuits also include an embedded security engine 1070.

[0090] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be used in a SOC integrated circuit 1000 to infer or predict operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0091] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0092] Figures 11A-11B Exemplary integrated circuits and associated graphics processors according to various embodiments described herein are illustrated, which may be fabricated using one or more IP cores. In addition to those illustrated, at least one embodiment may include other logic and circuitry, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0093] Figures 11A-11B This is a block diagram illustrating an exemplary graphics processor used within a SoC according to embodiments described herein. Figure 11AAn exemplary graphics processor 1110, which can be fabricated using one or more IP cores according to at least one embodiment, is shown. Figure 11B An additional exemplary graphics processor 1140, which can be fabricated using one or more IP cores, is shown according to at least one embodiment. In at least one embodiment, Figure 11A The graphics processor 1110 is a low-power graphics processor core. In at least one embodiment, Figure 11B The graphics processor 1140 is a higher-performance graphics processor core. In at least one embodiment, each graphics processor 1110, 1140 may be... Figure 9 A variant of the computer system 900.

[0094] In at least one embodiment, the graphics processor 1110 includes a vertex processor 1105 and one or more fragment processors 1115A-1115N (e.g., 1115A, 1115B, 1115C, 1115D to 1115N-1 and 1115N). In at least one embodiment, the graphics processor 1110 may execute different shader programs via separate logic, such that the vertex processor 1105 is optimized to perform operations for vertex shader programs, while one or more fragment processors 1115A-1115N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. In at least one embodiment, the vertex processor 1105 performs the vertex processing stage of the 3D graphics pipeline and generates primitive and vertex data. In at least one embodiment, one or more fragment processors 1115A-1115N use the primitive and vertex data generated by the vertex processor 1105 to generate framebuffers for display on a display device. In at least one embodiment, one or more fragment processors 1115A-1115N are optimized to execute fragment shader programs as provided in the OpenGL API, which can be used to perform operations similar to those of pixel shader programs provided in the Direct 3D API.

[0095] In at least one embodiment, the graphics processor 1110 additionally includes one or more memory management units (MMUs) 1120A-1120B, one or more caches 1125A-1125B, and one or more circuit interconnects 1130A-1130B. In at least one embodiment, the one or more MMUs 1120A-1120B provide virtual-to-physical address mapping for the graphics processor 1110 (including for the vertex processor 1105 and / or fragment processors 1115A-1115N), and may reference vertex or image / texture data stored in memory in addition to vertex or image / texture data stored in the one or more caches 1125A-1125B. In at least one embodiment, the one or more MMUs 1120A-1120B may be synchronized with other MMUs within the system, including with... Figure 11A One or more application processors 1105, graphics processors 1115, and / or video processors 1120 are associated with one or more MMUs, such that each processor 1105-1120 can participate in a shared or unified virtual memory system. In at least one embodiment, one or more circuit interconnects 1130A-1130B enable the graphics processor 1110 to interface with other IP cores within the SoC via the SoC's internal bus or via a direct connection.

[0096] In at least one embodiment, the graphics processor 1140 includes, as shown below: Figure 11B The one or more shader cores 1155A-1155N (e.g., 1155A, 1155B, 1155C, 1155D, 1155E, 1155F to 1155N-1 and 1155N) shown provide a unified shader core architecture, wherein a single core or type or core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. In at least one embodiment, the number of shader cores can vary. In at least one embodiment, the graphics processor 1140 includes an inter-core task manager 1145, which acts as a thread dispatcher for dispatching execution threads to one or more shader cores 1155A-1155N and a tile unit 1158 to accelerate tile-based rendering operations, where scene rendering operations are subdivided in image space, for example, to utilize local spatial consistency within the scene or optimize the use of internal caches.

[0097] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0098] Figure 12This is a block diagram illustrating a computing system 1200 according to at least one embodiment. In at least one embodiment, the computing system 1200 includes a processing subsystem 1201 having one or more processors 1202 and a system memory 1204 communicating via an interconnect path that may include a memory hub 1205. In at least one embodiment, the memory hub 1205 may be a separate component within a chipset assembly or may be integrated within one or more processors 1202. In at least one embodiment, the memory hub 1205 is coupled to an I / O subsystem 1211 via a communication link 1206. In at least one embodiment, the I / O subsystem 1211 includes an I / O hub 1207 that enables the computing system 1200 to receive input from one or more input devices 1208. In at least one embodiment, the I / O hub 1207 enables a display controller to provide output to one or more display devices 1210A, the display controller being included in one or more processors 1202. In at least one embodiment, one or more display devices 1210A coupled to the I / O hub 1207 may include local, internal, or embedded display devices.

[0099] In at least one embodiment, the processing subsystem 1201 includes one or more parallel processors 1212 coupled to the memory hub 1205 via a bus or other communication link 1213. In at least one embodiment, the communication link 1213 may use one of any number of standards based on a communication link technology or protocol (such as, but not limited to, PCI Express), or may be a vendor-specific communication interface or communication architecture. In at least one embodiment, one or more parallel processors 1212 form a computationally concentrated parallel or vector processing system, which may include a large number of processing cores and / or processing clusters, such as integrated many-core (MIC) processors. In at least one embodiment, some or all of the parallel processors 1212 form a graphics processing subsystem that can output pixels to one or more display devices 1210A coupled via an I / O hub 1207. In at least one embodiment, one or more parallel processors 1212 may also include a display controller and a display interface (not shown) for implementing direct connection to one or more display devices 1210B. In at least one embodiment, the parallel processor 1212 includes one or more cores, such as the graphics core 1200 discussed herein.

[0100] In at least one embodiment, system storage unit 1214 may be connected to I / O hub 1207 to provide a storage mechanism for computing system 1200. In at least one embodiment, I / O switch 1216 may be used to provide an interface mechanism for enabling connectivity between I / O hub 1207 and other components, such as network adapter 1218 and / or wireless network adapter 1219 integrated into the platform, and various other devices that can be added via one or more additional devices 1220. In at least one embodiment, network adapter 1218 may be an Ethernet adapter or another wired network adapter. In at least one embodiment, wireless network adapter 1219 may include one or more of Wi-Fi, Bluetooth, near field communication (NFC), or other network devices including one or more wireless devices.

[0101] In at least one embodiment, the computing system 1200 may include other components, not explicitly shown, that may also be connected to the I / O hub 1207, including USB or other port connections, optical storage drives, video capture devices, etc. In at least one embodiment, the interconnect can be implemented using any suitable protocol, such as a PCI (Peripheral Component Interconnect) based protocol (e.g., PCI-Express) or other bus or point-to-point communication interface and / or protocol (e.g., NV-Link high-speed interconnect or interconnect protocols). Figure 12 The communication paths of each component.

[0102] In at least one embodiment, one or more parallel processors 1212 include circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constituting a graphics processing unit (GPU), such as one or more parallel processors 1212 including a graphics core 1200. In at least one embodiment, one or more parallel processors 1212 include circuitry optimized for general-purpose processing. In at least one embodiment, components of the computing system 1200 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, one or more parallel processors 1212, a memory hub 1205, one or more processors 1202, and an I / O hub 1207 may be integrated into a system-on-a-chip (SoC) integrated circuit. In at least one embodiment, components of the computing system 1200 may be integrated into a single package to form a system-in-package (SIP) configuration. In at least one embodiment, at least a portion of the components of the computing system 1200 may be integrated into a multi-chip module (MCM), which may interconnect with other MCMs to a modular computing system.

[0103] Inference and / or training logic 515 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 515 may be... Figure 12 The system is used for inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases as described herein.

[0104] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0105] processor

[0106] Figure 13A A parallel processor 1300 according to at least one embodiment is illustrated. In at least one embodiment, the various components of the parallel processor 1300 may be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In at least one embodiment, the illustrated parallel processor 1300 is according to an exemplary embodiment. Figure 12 Variations of the one or more parallel processors 1212 shown. In at least one embodiment, the parallel processor 1300 includes one or more graphics cores 1200.

[0107] In at least one embodiment, the parallel processor 1300 includes a parallel processing unit 1302. In at least one embodiment, the parallel processing unit 1302 includes an I / O unit 1304 that enables communication with other devices, including other instances of the parallel processing unit 1302. In at least one embodiment, the I / O unit 1304 can be directly connected to other devices. In at least one embodiment, the I / O unit 1304 is connected to other devices via a hub or switch interface (e.g., a memory hub 1305). In at least one embodiment, the connection between the memory hub 1305 and the I / O unit 1304 forms a communication link 1313. In at least one embodiment, the I / O unit 1304 is connected to a host interface 1306 and a memory crossbar switch 1316, wherein the host interface 1306 receives commands for performing processing operations, and the memory crossbar switch 1316 receives commands for performing memory operations.

[0108] In at least one embodiment, when host interface 1306 receives a command buffer via I / O unit 1304, host interface 1306 can route work operations for executing those commands to front end 1308. In at least one embodiment, front end 1308 is coupled to scheduler 1310 (which may be referred to as sequencer), which is configured to assign commands or other work items to processing cluster array 1312. In at least one embodiment, scheduler 1310 ensures that processing cluster array 1312 is correctly configured and in an active state before assigning tasks to clusters in processing cluster array 1312. In at least one embodiment, scheduler 1310 is implemented via firmware logic executed on a microcontroller. In at least one embodiment, the microcontroller-implemented scheduler 1310 can be configured to perform complex scheduling and work assignment operations at both coarse and fine granular levels, thereby enabling fast preemption and context switching of threads executing on processing cluster array 1312. In at least one embodiment, host software can demonstrate workloads for scheduling on processing cluster array 1312 via one of multiple graphics processing paths. In at least one embodiment, the workload can then be automatically distributed on the processing cluster array 1312 by the scheduler 1310 logic within the microcontroller, which includes the scheduler 1310.

[0109] In at least one embodiment, the processing cluster array 1312 may include up to "N" processing clusters (e.g., clusters 1314A, 1314B to 1314N), where "N" represents a positive integer (which may be an integer "N" different from the integers used in other diagrams). In at least one embodiment, each cluster 1314A-1314N of the processing cluster array 1312 may execute a large number of concurrent threads. In at least one embodiment, the scheduler 1310 may use various scheduling and / or work allocation algorithms to allocate work to clusters 1314A-1314N in the processing cluster array 1312, which may vary depending on the workload generated for each type of program or computation. In at least one embodiment, scheduling may be handled dynamically by the scheduler 1310, or may be partially assisted by compiler logic during the compilation of program logic configured to be executed by the processing cluster array 1312. In at least one embodiment, different clusters 1314A-1314N in the processing cluster array 1312 may be assigned to process different types of programs or to perform different types of computations.

[0110] In at least one embodiment, the processing cluster array 1312 can be configured to perform various types of parallel processing operations. In at least one embodiment, the processing cluster array 1312 is configured to perform general-purpose parallel computing operations. For example, in at least one embodiment, the processing cluster array 1312 may include logic for performing processing tasks, including filtering video and / or audio data, performing modeling operations, including physical operations, and performing data transformations.

[0111] In at least one embodiment, the processing cluster array 1312 is configured to perform parallel graphics processing operations. In at least one embodiment, the processing cluster array 1312 may include additional logic for supporting the execution of such graphics processing operations, including but not limited to texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. In at least one embodiment, the processing cluster array 1312 may be configured to execute shader programs related to graphics processing, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. In at least one embodiment, the parallel processing unit 1302 may transfer data from system memory via I / O unit 1304 for processing. In at least one embodiment, during processing, the transferred data may be stored in on-chip memory (e.g., parallel processor memory 1322) and then written back to system memory.

[0112] In at least one embodiment, when the parallel processing unit 1302 is used to perform graphics processing, the scheduler 1310 can be configured to divide the processing workload into tasks of approximately equal size to better distribute graphics processing operations among multiple clusters 1314A-1314N in the processing cluster array 1312. In at least one embodiment, portions of the processing cluster array 1312 can be configured to perform different types of processing. For example, in at least one embodiment, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display. In at least one embodiment, intermediate data generated by one or more of the clusters 1314A-1314N can be stored in a buffer to allow intermediate data to be transferred between the clusters 1314A-1314N for further processing.

[0113] In at least one embodiment, the processing cluster array 1312 may receive processing tasks to be executed via a scheduler 1310, which receives commands defining the processing tasks from a front end 1308. In at least one embodiment, the processing task may include an index of data to be processed, such as surface (patch) data, raw data, vertex data, and / or pixel data, as well as state parameters and commands defining how the data is processed (e.g., what program to execute). In at least one embodiment, the scheduler 1310 may be configured to acquire an index corresponding to a task, or may receive an index from the front end 1308. In at least one embodiment, the front end 1308 may be configured to ensure that the processing cluster array 1312 is configured to be active before initiating the workload specified by an incoming command buffer (e.g., a batch buffer, push buffer, etc.).

[0114] In at least one embodiment, each of one or more instances of the parallel processing unit 1302 may be coupled to the parallel processor memory 1322. In at least one embodiment, the parallel processor memory 1322 may be accessed via a memory crossbar switch 1316, which may receive memory requests from the processing cluster array 1312 and the I / O unit 1304. In at least one embodiment, the memory crossbar switch 1316 may be accessed via a memory interface 1318. In at least one embodiment, the memory interface 1318 may include a plurality of partition units (e.g., partition units 1320A, 1320B to 1320N), each of which may be coupled to a portion (e.g., a memory cell) of the parallel processor memory 1322. In at least one embodiment, the number of partition units 1320A-1320N is configured to be equal to the number of memory units, such that the first partition unit 1320A has a corresponding first memory unit 1324A, the second partition unit 1320B has a corresponding second memory unit 1324B, and the Nth partition unit 1320N has a corresponding Nth memory unit 1324N. In at least one embodiment, the number of partition units 1320A-1320N may not be equal to the number of memory units.

[0115] In at least one embodiment, memory cells 1324A-1324N may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In at least one embodiment, memory cells 1324A-1324N may also include 3D stacked memory, including but not limited to high-bandwidth memory (HBM), HBM2e, and HDM3. In at least one embodiment, rendering targets such as framebuffers or texture maps can be stored across memory cells 1324A-1324N, allowing partitioning cells 1320A-1320N to write portions of each rendering target in parallel, to efficiently utilize the available bandwidth of the parallel processor memory 1322. In at least one embodiment, local instances of the parallel processor memory 1322 may be excluded to facilitate a unified memory design that utilizes system memory and local cache memory.

[0116] In at least one embodiment, any of clusters 1314A-1314N in the processing cluster array 1312 can process data to be written to any memory cell 1324A-1324N within the parallel processor memory 1322. In at least one embodiment, the memory crossbar switch 1316 can be configured to transfer the output of each cluster 1314A-1314N to any partition cell 1320A-1320N or another cluster 1314A-1314N, which can perform additional processing operations on the output. In at least one embodiment, each cluster 1314A-1314N can communicate with the memory interface 1318 via the memory crossbar switch 1316 to read from or write to various external memory devices. In at least one embodiment, the memory crossbar switch 1316 has a connection to a memory interface 1318 for communicating with I / O unit 1304, and a connection to a local instance of parallel processor memory 1322, enabling processing units within different processing clusters 1314A-1314N to communicate with system memory or other memory not local to parallel processing unit 1302. In at least one embodiment, the memory crossbar switch 1316 can use virtual channels to separate traffic flows between clusters 1314A-1314N and partition units 1320A-1320N.

[0117] In at least one embodiment, multiple instances of the parallel processing unit 1302 may be provided on a single add-in card, or multiple add-in cards may be interconnected. In at least one embodiment, different instances of the parallel processing unit 1302 may be configured to interoperate, even if the different instances have different numbers of processing cores, different numbers of local parallel processor memories, and / or other configuration differences. For example, in at least one embodiment, some instances of the parallel processing unit 1302 may include higher-precision floating-point units relative to other instances. In at least one embodiment, a system including one or more instances of the parallel processing unit 1302 or the parallel processor 1300 may be implemented in various configurations and form factors, including but not limited to desktop computers, laptop or handheld personal computers, servers, workstations, game consoles, and / or embedded systems.

[0118] Figure 13B This is a block diagram of a partitioning unit 1320 according to at least one embodiment. In at least one embodiment, the partitioning unit 1320 is... Figure 13A This is an example of one of the partitioning units 1320A-1320N. In at least one embodiment, the partitioning unit 1320 includes an L2 cache 1321, a frame buffer interface 1325, and a ROP 1326 (raster operation unit). In at least one embodiment, the L2 cache 1321 is a read / write cache configured to perform load and store operations received from the memory crossbar switch 1316 and the ROP 1326. In at least one embodiment, the L2 cache 1321 outputs read misses and urgent write-back requests to the frame buffer interface 1325 for processing. In at least one embodiment, updates can also be sent to the frame buffer for processing via the frame buffer interface 1325. In at least one embodiment, the frame buffer interface 1325 communicates with memory cells in the parallel processor memory (such as...). Figure 13A It is coupled to one of the memory cells 1324A-1324N (e.g., within the parallel processor memory 1322).

[0119] In at least one embodiment, ROP 1326 is a processing unit that performs raster operations such as stenciling, z-testing, blending, etc. In at least one embodiment, ROP 1326 then outputs processed graphics data stored in graphics memory. In at least one embodiment, ROP 1326 includes compression logic for compressing depth or color data written to memory and decompressing depth or color data read from memory. In at least one embodiment, the compression logic may be lossless compression logic utilizing one or more of a variety of compression algorithms. In at least one embodiment, the type of compression performed by ROP 1326 may vary based on the statistical characteristics of the data to be compressed. For example, in at least one embodiment, incremental color compression is performed on depth and color data per tile.

[0120] In at least one embodiment, ROP 1326 is included within each processing cluster (e.g., Figure 13A The clusters 1314A-1314N are used instead of the partition unit 1320. In at least one embodiment, read and write requests for pixel data, rather than pixel fragment data, are transmitted via the memory crossbar switch 1316. In at least one embodiment, the processed graphics data can be displayed on a display device (such as...) Figure 12 Displayed on one or more display devices 1210, routed by processor 1302 for further processing, or by... Figure 13A One of the processing entities within the parallel processor 1300 is routed for further processing.

[0121] Figure 14 This is a processing system according to at least one embodiment. In at least one embodiment, system 1400 includes one or more processors 1402 and one or more graphics processors 1408, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1402 or processor cores 1407. In at least one embodiment, system 1400 is a processing platform included within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices. In at least one embodiment, one or more graphics processors 1408 include one or more graphics cores 1200.

[0122] In at least one embodiment, system 1400 may include or be integrated into a server-based gaming platform, a game console including a game and media console, a mobile game console, a handheld game console, or an online game console. In at least one embodiment, system 1400 is a mobile phone, smartphone, tablet computing device, or mobile internet device. In at least one embodiment, processing system 1400 may also include components coupled to or integrated into a wearable device, such as a smartwatch wearable device, smart glasses device, augmented reality device, or virtual reality device. In at least one embodiment, processing system 1400 is a television or set-top box device having one or more processors 1402 and a graphical interface generated by one or more graphics processors 1408.

[0123] In at least one embodiment, each of the one or more processors 1402 includes one or more processor cores 1407 for processing instructions that, when executed, perform operations against the system and user software. In at least one embodiment, each of the one or more processor cores 1407 is configured to process a specific instruction sequence 1409. In at least one embodiment, the instruction sequence 1409 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). In at least one embodiment, each processor core 1407 may process a different instruction sequence 1409, which may include instructions that facilitate the emulation of other instruction sequences. In at least one embodiment, the processor core 1407 may also include other processing devices, such as a digital signal processor (DSP).

[0124] In at least one embodiment, processor 1402 includes cache memory 1404. In at least one embodiment, processor 1402 may have a single internal cache or more levels of internal caches. In at least one embodiment, the cache memory is shared among various components of processor 1402. In at least one embodiment, processor 1402 also uses an external cache (e.g., a Level 3 (L3) cache or a last-level cache (LLC)) (not shown), which can be shared among processor cores 1407 using known cache coherence techniques. In at least one embodiment, processor 1402 additionally includes a register file 1406, which may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, register file 1406 may include general-purpose registers or other registers.

[0125] In at least one embodiment, one or more processors 1402 are coupled to one or more interface buses 1410 to transmit communication signals, such as address, data, or control signals, between the processors 1402 and other components in the system 1400. In at least one embodiment, the interface bus 1410 may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus 1410 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, one or more processors 1402 include an integrated memory controller 1416 and a platform controller hub 1430. In at least one embodiment, the memory controller 1416 facilitates communication between memory devices and other components of the system 1400, while the platform controller hub (PCH) 1430 provides connectivity to I / O devices via a local I / O bus.

[0126] In at least one embodiment, memory device 1420 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or some other memory device with suitable performance for use as processor memory. In at least one embodiment, memory device 1420 may operate as system memory of system 1400 for storing data 1422 and instructions 1421 for use when one or more processors 1402 execute an application or process. In at least one embodiment, memory controller 1416 is also coupled to an optional external graphics processor 1412, which may communicate with one or more graphics processors 1408 of processor 1702 to perform graphics and media operations. In at least one embodiment, display device 1411 may be connected to one or more processors 1402. In at least one embodiment, display device 1411 may include one or more internal display devices, such as in mobile electronic devices or laptop devices, or external display devices attached via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 1411 may include a head-mounted display (HMD), such as a stereoscopic display device for virtual reality (VR) or augmented reality (AR) applications.

[0127] In at least one embodiment, the platform controller hub 1430 enables peripheral devices to connect to the memory device 1420 and the processor 1402 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 1446, a network controller 1434, a firmware interface 1428, a wireless transceiver 1426, a touch sensor 1425, and a data storage device 1424 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1424 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCIe). In at least one embodiment, the touch sensor 1425 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1426 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or LTE transceiver. In at least one embodiment, the firmware interface 1428 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, network controller 1434 can implement network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to interface bus 1410. In at least one embodiment, audio controller 1446 is a multi-channel high-definition audio controller. In at least one embodiment, system 1400 includes an optional legacy I / O controller 1440 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to system 1400. In at least one embodiment, platform controller hub 1430 can also be connected to one or more Universal Serial Bus (USB) controllers 1442, which connect input devices such as keyboard and mouse 1443 combinations, camera 1444, or other USB input devices.

[0128] In at least one embodiment, instances of the memory controller 1416 and platform controller hub 1430 may be integrated into a discrete external graphics processor, such as external graphics processor 1412. In at least one embodiment, the platform controller hub 1430 and / or the memory controller 1416 may be external to one or more processors 1402. For example, in at least one embodiment, system 1400 may include an external memory controller 1416 and platform controller hub 1430, which may be configured as a memory controller hub and peripheral controller hub in a system chipset communicating with one or more processors 1402.

[0129] The embodiments presented herein can perform inference orchestration in a distributed resource environment and allow for network acceleration.

[0130] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0131] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar pronouns, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (wherein unmodified, it refers to a physical connection) should be interpreted as partially or wholly included, attached to, or connected together, even with some intervening elements. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0132] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of each of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” indicates multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0133] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) executed jointly by hardware or a combination thereof on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within a transceiver of transient signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises a plurality of non-transitory computer-readable storage media, and one or more of the various non-transitory storage media lack the complete code, but the plurality of non-transitory computer-readable storage media collectively store the complete code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0134] In at least one embodiment, an arithmetic logic unit is a set of combinational logic circuits that takes one or more inputs to produce a result. In at least one embodiment, a processor uses arithmetic logic units to implement mathematical operations, such as addition, subtraction, or multiplication. In at least one embodiment, arithmetic logic units are used to implement logical operations, such as logical AND / OR or XOR. In at least one embodiment, arithmetic logic units are stateless and made of physical switching elements, such as semiconductor transistors arranged to form logic gates. In at least one embodiment, arithmetic logic units may operate internally as stateful logic circuits with an associated clock. In at least one embodiment, arithmetic logic units may be configured as asynchronous logic circuits whose internal state is not maintained in an associated set of registers. In at least one embodiment, a processor uses arithmetic logic units to combine operands stored in one or more registers of the processor and produce an output that can be stored by the processor in another register or memory location.

[0135] In at least one embodiment, as a result of processing instructions retrieved by the processor, the processor presents one or more inputs or operands to the arithmetic logic unit (ALU), causing the ALU to produce a result at least partially based on instruction code provided to the ALU of the inputs. In at least one embodiment, the instruction code provided by the processor to the ALU is at least partially based on instructions executed by the processor. In at least one embodiment, combinational logic in the ALU processes the inputs and produces an output placed on a bus within the processor. In at least one embodiment, the processor selects a destination register, memory location, output device, or output storage location on the output bus to clock the processor such that the result produced by the ALU is sent to the desired location.

[0136] Within the scope of this application, the term Arithmetic Logic Unit or ALU is used to refer to any computational logic circuit that processes operands to produce results. For example, in this document, the term ALU may refer to a floating-point unit, a DSP, a tensor core, a shader core, a coprocessor, or a CPU.

[0137] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the processes described herein individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the performance of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating differently, such that the distributed computer system performs the operations described herein, and that no single device performs all operations.

[0138] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not impose a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0139] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference to the same extent that each reference is individually and specifically indicated as incorporated herein by reference and its entire contents are set forth herein.

[0140] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0141] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “calculation,” “operation,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical quantities (e.g., electronic quantities) in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0142] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0143] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0144] While this document describes example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0145] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not limited to the specific features or actions described, but rather discloses specific features and actions as exemplary forms for implementing the claims.

Claims

1. A system comprising: One or more processors are used for: Receive a request to perform at least one inference operation; The inference orchestrator is used to determine a sequence of tasks to be performed on the at least one inference operation, the tasks including: obtaining context information from at least one context node, and performing the at least one inference operation using at least one inference node; Append metadata indicating the order to which the request should be forwarded to the at least one context node and the at least one inference node to be ordered to the request; The request with the additional metadata is forwarded to the at least one context node and the at least one inference node according to the ordering and independently of the inference operation, the at least one context node providing the context information for the at least one inference node to perform the at least one inference operation; and In response to receiving one or more results of the at least one inference operation, the inference orchestrator determines the response to be sent regarding the request.

2. The system of claim 1, wherein the inference orchestrator includes a programmable network device.

3. The system of claim 2, wherein the programmable network device is a data processing unit (DPU) or a smart network interface card (SmartNIC).

4. The system of claim 1, wherein the sorting is determined in part based on the category of the request as determined by the inference orchestrator.

5. The system of claim 1, wherein the request includes a prompt to generate one or more types of content, the one or more types of content including at least text, images, audio, video, or animated content.

6. The system of claim 1, wherein the at least one context node and the at least one inference node comprise a physical computing device or a virtual computing device.

7. The system of claim 1, wherein the inference orchestrator is capable of receiving multiple inference operation results from at least one inference node, and is capable of at least selecting or combining the results to be returned along with the response.

8. The system of claim 1, wherein the at least one inference node is further selected based on at least one preference of the user associated with the request.

9. The system of claim 1, wherein the at least one context node is further selected in part based on one or more types of context information to be obtained, wherein the context information can include public or private information in part based on the type or category of the context information.

10. The system of claim 1, wherein if at least one context node, at least one inference node, or at least one processing node lacks a network programmable device for updating the request and sequentially forwarding the request to the next node, the inference orchestrator is capable of receiving a version of the request prior to the final version of the request.

11. The system of claim 1, wherein the metadata further includes information about the target recipient of the response, the response being permitted to be sent by the final inference node or the inference orchestrator.

12. At least one processor, comprising: Processing circuitry system, used for: The inference orchestrator determines a sequence of tasks to be performed on at least one inference operation determined in response to a received inference request, the tasks including: obtaining context information from at least one context node, and performing the at least one inference operation using at least one inference node; Append metadata indicating the order to which the request should be forwarded to the at least one context node and the at least one inference node to be ordered to the request; The request with the additional metadata is forwarded to the at least one context node and the at least one inference node according to the ordering and independently of the inference operation, the at least one context node providing the context information for the at least one inference node to perform the at least one inference operation; and In response to receiving one or more results of the at least one inference operation, the inference orchestrator determines the response to be sent regarding the request.

13. At least one processor as claimed in claim 12, wherein the sorting is determined in part based on the category of the request determined by the inference orchestrator.

14. The at least one processor of claim 12, wherein the inference orchestrator is capable of receiving multiple operation results from at least one inference node or at least one processing node, and is capable of at least selecting or combining the results to be returned together with the response.

15. The at least one processor of claim 12, wherein if at least one context node, at least one inference node, or at least one processing node lacks a network programmable device for updating the request and sequentially forwarding the request to the next node, the inference orchestrator is capable of receiving a version of the request prior to the final version of the request.

16. An inference orchestrator, comprising: At least one processor is configured to determine a sequence of context nodes and inference nodes to process a received inference request, and to append metadata indicating the sequence of context nodes and inference nodes to the inference request, wherein the context nodes and the inference nodes have one or more programmable network devices that allow parsing the inference request, updating the inference request, and forwarding the inference request to any subsequent node in the sequence without returning the inference request to the inference orchestrator during the sequence.

17. The inference orchestrator of claim 16, wherein the ordering is determined in part based on the category of the request determined by the inference orchestrator.

18. The inference orchestrator of claim 16, wherein the inference orchestrator is capable of receiving multiple inference operation results from at least one inference node, and is capable of at least selecting or combining the results to be returned along with the response.

19. The inference orchestrator of claim 16, wherein if at least one context node, at least one inference node, or at least one processing node lacks a network programmable device for updating the request and sequentially forwarding the request to the next node, the inference orchestrator is capable of receiving a version of the request prior to the final version of the request.

20. The inference orchestrator of claim 16, wherein the inference orchestrator is implemented in at least one of the following: A system used to perform simulation operations; A system used to perform simulations to test or validate autonomous machine applications; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system used for rendering graphics output; A system used to perform deep learning operations; A system for performing generative AI operations using large language model LLM; Systems implemented using edge devices; Systems used to generate or present virtual reality (VR) content; A system for generating or presenting augmented reality (AR) content; A system for generating or presenting mixed reality (MR) content; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; A system for performing hardware tests using simulation; A system for performing generative operations using a language model LM; Systems for generating synthetic data; A collaborative content creation platform for 3D assets; or A system that utilizes cloud computing resources at least in part.

Images