Metric and security-based accelerator service rescheduling and autoscaling using programmable network devices

Abstract

To provide metrics and security-based accelerator service rescheduling and auto-scaling using a programmable network device.SOLUTION: An apparatus includes processors to: collect metrics corresponding to communication links between microservices of a service managed by a service mesh; determine, based on analysis of the metrics, that a workload can be accelerated by offloading a hardware accelerator device; cause a programmable network device to allocate the hardware accelerator device to a cluster; register the hardware accelerator device with the service mesh; and schedule the workload of the service to the hardware accelerator device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Embodiments generally relate to data processing, and more specifically to metric and security-based accelerator service rescheduling and auto-scaling using programmable network devices.

Background Art

[0002] Distributed computing is increasing in data centers. Cloud Service Providers (CSPs) are deploying solutions where workload processing is distributed across distributed computing resources such as CPUs, GPUs, and hardware accelerators (including Field Programmable Gate Arrays (FPGAs)) that are connected via a network instead of being on the same platform, and connected via physical links such as Peripheral Component Interconnect Express (PCIe). Distributed computing enables improved resource utilization by allowing more efficient use of available resources, reducing total cost of ownership. Distributed computing also enables pooling of a large number of hardware accelerators for large-scale computations, allowing computations to be performed more efficiently and better.

[0003] In a distributed computing environment, a typical worker node in a compute cluster can handle hundreds of container workloads simultaneously. These worker nodes can also have statically attached, dedicated hardware accelerators optimized for computationally intensive tasks. For example, different classes of hardware accelerators can be optimized to efficiently perform encryption and compression algorithms such as Transport Layer Security (TLS) or zlib, respectively. However, static hardware accelerator resources are not always available for all workloads running on the worker nodes due to resource scarcity. To accelerate computationally intensive operations in a meaningful way, hardware accelerator resources cannot be distributed too thinly to guarantee quality of service (QoS).

[0004] To allow for a detailed understanding of the above-described features of this embodiment, a more specific description of the embodiment, which has been briefly summarized above, can be obtained by reference to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings illustrate typical embodiments and should therefore not be considered limiting in scope. The drawings are not to scale. Generally, the same reference numerals are used throughout the drawings and accompanying descriptive text to refer to the same or similar parts. [Brief explanation of the drawing]

[0005] [Figure 1] This disclosure describes a computing device that uses an XPU scheduler extender to provide metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, as implemented in this disclosure.

[0006] [Figure 2] Figure 1 shows the XPU scheduler extender in one implementation form of this disclosure.

[0007] [Figure 3] This disclosure describes a data center system that provides metric and security-based accelerator service rescheduling and autoscaling using programmable network devices, as implemented in this form.

[0008] [Figure 4] A block diagram of a data center system 400 that implements metric and security-based accelerator service rescheduling and autoscaling using programmable network devices, as implemented in the form of this disclosure, is shown.

[0009] [Figure 5] This flowchart illustrates an embodiment of a method for metric and security-based accelerator service rescheduling and autoscaling using programmable network devices.

[0010] [Figure 6] This flowchart illustrates an embodiment of a method for implementing security measures while performing metric and security-based accelerator service rescheduling and autoscaling using programmable network devices.

[0011] [Figure 7] This is a schematic diagram of an exemplary electronic computing device for enabling metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, according to several embodiments. [Modes for carrying out the invention]

[0012] The implementations of this disclosure describe metric and security-based accelerator service rescheduling and autoscaling using programmable network devices. Cloud service providers (CSPs) deploy solutions to data centers where workload processing is distributed across distributed computing resources such as central processing units (CPUs), graphics processing units (GPUs), and / or hardware accelerators (including, but not limited to, field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)) that are connected via a network (rather than being on the same platform and connected via physical links such as Peripheral Interconnection Express (PCIe)). Distributed computing enables improved resource utilization and reduces the cost of ownership by allowing for more efficient use of available resources. Distributed computing also enables pooling a large number of hardware accelerators for large-scale computations, making computations more efficient and better performed.

[0013] Service providers and enterprises are investing heavily in hyperscale data centers to provide efficient computing for cloud-native applications and microservices. Applications providing these services need access to high-speed, low-latency storage and secure networking infrastructure. Distributed computing solutions can provide such high-speed, low-latency storage and secure networking infrastructure. However, infrastructure services such as virtual switching, security, and storage can consume a significant number of CPU cycles.

[0014] Programmable network devices can be used to accelerate network infrastructure and free up CPU cores to improve application performance. Programmable network devices may include, for example, Infrastructure Processing Units (IPUs), Data Processing Units (DPUs), and Smart Network Interface Cards (SmartNICs). While this specification may refer to programmable network devices as IPUs, it is not intended to limit programmable network devices to such implementations and may encompass other implementations. Programmable network devices such as IPUs improve data center utilization by enabling cloud service providers to customize infrastructure function deployments at software speed while allowing for flexible workload placement. Programmable network devices can intelligently manage system-level resources by securely accelerating data center networking and storage infrastructure functions.

[0015] Data centers with distributed computing can further leverage microservices architecture to accelerate network infrastructure services. A microservices architecture can represent an application as a loosely coupled collection of microservices, which can refer to processes that communicate over a network to achieve their goals using technology-independent protocols such as HTTP or gRPC. In some cases, microservices may be deployed using a container orchestration platform that provides containerized workloads and / or services. A container orchestration platform can leverage a service mesh to manage the large volume of network-based inter-process communication between microservices. A service mesh is a dedicated software infrastructure layer for microservices that includes elements that enable communication between microservices to be fast, reliable, and secure. A service mesh provides capabilities including service discovery, load balancing, encryption, observability, traceability, and authentication and authorization.

[0016] In a distributed computing environment, a typical worker node in a microservices architecture compute cluster can handle hundreds of container workloads simultaneously. These worker nodes can also have statically mounted, dedicated hardware accelerators optimized for computationally intensive tasks. (As described herein, hardware accelerators may also be called hardware accelerator devices, hardware accelerator resources, accelerator resources, or accelerators.) For example, classes of hardware accelerators can be optimized to efficiently perform encryption and compression algorithms such as Transport Layer Security (TLS) or zlib, respectively. However, static hardware accelerator resources are not typically available for all workloads running on worker nodes due to resource scarcity. To accelerate computationally intensive operations in a meaningful way, hardware accelerator resources cannot be distributed too thinly to guarantee quality of service (QoS).

[0017] In the relevant context, the control plane scheduler of a container orchestration platform (which manages a microservices architecture) can perceive hardware accelerator virtual functions (VFs) as "extended resources." The number of VFs available on a given accelerator may be limited. For example, in the case of a cryptographic accelerator card, there may be three physical accelerator engines, each exposing 16 VFs, resulting in 48 possible extended and assignable resources per node. A container application in a microservices architecture may request one or more such accelerator resources, and after a node's accelerator resources are exhausted, the control plane scheduler will not schedule workloads that request such accelerator resources on compute nodes (e.g., server CPUs), even if the compute node has available compute resources. This can lead to underutilization of compute nodes.

[0018] In some cases, containers can benefit from using accelerator resources, but can also run without trade-offs such as performance degradation and increased CPU utilization, or may be less affected by which accelerator resource is used (e.g., FPGA vs. GPU vs. ASIC). In these cases, traditional solutions cannot handle them efficiently. Furthermore, if a static cluster completely exhausts a particular type of accelerator resource, new workloads remain pending until the resource is released again. This increases latency and degrades the performance of microservice architectures.

[0019] Traditional methods for scheduling augmented resources in microservice architectures include those used by the control plane scheduler of container orchestration platforms, as follows: When a workload requests augmented resources, it is scheduled to the compute node that provides such resources, and the node's available resource count is decremented. The available resources for each node are static and are reported to the cluster's control plane scheduler during initialization. To overcome the limitations of workloads operating in a pending state, one traditional method can provide an "infinite" number of virtual accelerator resources, with a subset of them including the actual accelerator hardware backend. However, this method does not address the situation when all of the "actual" accelerator hardware backend resources are fully utilized.

[0020] Metric-based scheduling in cloud orchestrators is not used in traditional methods to deploy hardware accelerator resources, particularly XPU resources, which can be distributed. XPU refers to heterogeneous and / or cross-architecture ("X") processing units, including hardware accelerators (e.g., CPUs, GPUs, FPGAs, ASICs, inference accelerators, cryptographic accelerators, and other dedicated hardware accelerators). For example, the Kubernetes project operates as a workload "vertical autoscaler." The concept is to monitor workload performance and dynamically add CPU time and / or memory to the workload if its performance degrades. However, this approach focuses on increasing or decreasing native (CPU and memory) resources and does not consider whether certain compute-intensive tasks can be offloaded to dedicated accelerators (e.g., XPUs) and / or coprocessors.

[0021] Other container orchestrator service mesh solutions work by having a sidecar proxy with every container application. If such a sidecar proxy uses hardware accelerators for common tasks such as Transport Layer Security (TLS) handshake accelerators or Hypertext Transport Protocol (HTTP) compression, then, due to the static limit of 48 resources (continuing the previous example), the application limit becomes 48 pods, including the sidecar proxy along with the application container. Thus, this scheduling mechanism directly contributes to resource scarcity problems, as workloads must wait for accelerator resources to become available before they can be scheduled.

[0022] Furthermore, if some of the granted resources are "empty" (i.e., do not include actual hardware accelerator access) but are granted to enable scheduling of more than 48 applications, predictability is lost (e.g., there is no way to know whether a service can meet performance goals). In such cases, a fully optimized microservice graph can observe bottlenecks in the flow, and since the traffic volume is already reduced in the previous part of the microservice flow graph, other hardware accelerator resources are prevented from being fully utilized.

[0023] Implementations of the present disclosure address the above technical drawbacks by providing metric and security-based accelerator service rescheduling and auto-scaling using programmable network devices. The implementations herein provide techniques for utilizing workload telemetry data, programmable network devices (such as IPUs), and XPU (x processing unit) hardware accelerators (e.g., CPUs, GPUs, FPGAs, ASICs, inference accelerators, cryptographic accelerators, other dedicated hardware accelerators, etc.) to achieve scalable and more dynamic computing resource allocation to workloads that benefit from acceleration in a computing cluster.

[0024] The implementations provide an XPU scheduler extender that offers two usage models, and a cluster node agent. In the implementations herein, the XPU scheduler extender may also be referred to as a scheduler extender or a scheduler extender circuit. In the first usage model, the XPU scheduler extender communicates with a main cluster orchestrator and the cluster node agent to relay information related to scaling of the amount of XPU accelerator computing resources to the cluster. In the second usage model, the XPU scheduler communicates with the data center cluster node agent to configure more computing bandwidth for workloads that are already consuming XPU resources, and / or switch the underlying XPU type for more efficient computing.

[0025] In the implementations herein, the XPU scheduler extender collects and analyzes metrics from workloads. The metrics can be mapped to functions that may also be accelerated on the XPU. For example, the number of TLS handshakes per second, or the amount of bytes of data compressed on the CPU, can be mapped to an encryption / compression accelerator. Using metric-based XPU scheduling, an optimized amount of XPU resources can be made available to the cluster, assigned to workloads that utilize them, and workloads that do not benefit from them can be scheduled to run on the CPU.

[0026] The implementation of this disclosure offers technical advantages over the conventional methods described above. One technical advantage is that XPU accelerator resources are better targeted because they are dynamically used to remove bottlenecks in the microservice flow graph and an optimized amount of XPU accelerator resources is implemented. Another technical advantage is that XPU accelerator resources are not moved to containers at usage levels so low that the resulting overhead degrades application performance. A further technical advantage is that nodes do not exhaust XPU accelerator resources as easily because the maximum number of containers running on a node is no longer determined by the available accelerator resources (e.g., in a service mesh use case).

[0027] Figure 1 shows a computing device 100 employing an XPU scheduler extender 110 to provide metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, according to an implementation of the present disclosure. Computing device 100 represents (but does not include) communication and data processing devices, including but not limited to smart voice command devices, intelligent personal assistants, home / office automation systems, home appliances (e.g., washing machines, television sets, etc.), mobile devices (e.g., smartphones, tablet computers, etc.), gaming devices, handheld devices, wearable devices (e.g., smartwatches, smart bracelets, etc.), virtual reality (VR) devices, head-mounted displays (HMDs), Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, set-top boxes (e.g., internet-based cable television set-top boxes, etc.), global positioning system (GPS) based devices, and automotive infotainment devices.

[0028] In some embodiments, the computing device 100 includes, operates with, is embedded in, or facilitates any number and types of other smart devices, such as (but not limited to) autonomous machines or artificial intelligence agents, including mechanical agents or machines, electronic agents or machines, virtual agents or machines, electromechanical agents or machines, etc. Examples of autonomous machines or artificial intelligence agents include (but not limited to) robots, autonomous vehicles (e.g., self-driving cars, self-flying aircraft, self-sailing boats, etc.), autonomous equipment, autonomous construction vehicles, autonomous medical devices, etc. Furthermore, “autonomous vehicles” are not limited to automobiles and can include any number and types of autonomous machines, such as robots, autonomous equipment, and autonomous household devices, and any one or more tasks or operations associated with such autonomous machines may be referred to interchangeably with autonomous driving.

[0029] Furthermore, for example, computing device 100 may include a computer platform that hosts an integrated circuit ("IC"), such as a system-on-a-chip ("SOC" or "SoC"), which integrates various hardware and / or software components of computing device 100 onto a single chip.

[0030] As shown in the figure, in one embodiment, the computing device 100 may include (but is not limited to) a graphics processing unit ("GPU" or simply a "graphics processor") 114, a graphics driver (also called a "GPU driver," "graphics driver logic," "driver logic," user-mode driver (UMD), user-mode driver framework (UMDF), or simply a "driver") 115, a central processing unit ("CPU" or simply an "application processor") 112, memory 108, network devices, drivers, etc., as well as any number and types of hardware and / or software components such as input / output (I / O) sources 104, such as a touchscreen, touch panel, touchpad, virtual or regular keyboard, virtual or regular mouse, ports, connectors, etc. The computing device 100 may also include an operating system (OS) 106 that acts as an interface between the hardware and / or physical resources of the computing device 100 and the user.

[0031] It should be understood that in certain implementations, systems with fewer or more components than those in the examples above may be available. Therefore, the configuration of computing device 100 may vary from implementation to implementation depending on numerous factors such as price constraints, performance requirements, technological improvements, or other circumstances.

[0032] The embodiments may be implemented as one or more microchips or integrated circuits interconnected using a parent board, hardwired logic, software stored in memory devices and executed by a microprocessor, firmware, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), or any combination thereof. The terms “logic,” “module,” “component,” “engine,” “circuit,” “element,” and “mechanism” may include, for example, software, hardware, and / or combinations thereof, such as firmware.

[0033] In one embodiment, as shown in the figure, the XPU scheduler extender 110 may be hosted by memory 108 that communicates with I / O sources 104 such as microphones and speakers of the computing device 100. In another embodiment, the XPU scheduler extender 110 may be part of the operating system 106 or hosted by the operating system 106. In yet another embodiment, the XPU scheduler extender 110 may be hosted or facilitated by a graphics driver 115. In yet another embodiment, the XPU scheduler extender 110 may be hosted by or part of a hardware accelerator 114. For example, the XPU scheduler extender 110 may be incorporated into or implemented as part of the processing hardware of the hardware accelerator 114, such as in the form of an XPU scheduler extender 140. In yet another embodiment, the XPU scheduler extender 110 may be hosted by or part of a graphics processing unit ("GPU" or simply "graphics processor") 116 or the firmware of the graphics processor 116. For example, the XPU scheduler extender may be incorporated into or implemented as part of the processing hardware of the graphics processor 116, such as in the form of an XPU scheduler extender 130. Similarly, in yet another embodiment, the XPU scheduler extender 110 may be hosted by or part of a central processing unit ("CPU" or simply "application processor") 112. For example, the XPU scheduler extender 110 may be incorporated into or implemented as part of the processing hardware of the application processor 112, such as in the form of an XPU scheduler extender 120.In some embodiments, the XPU scheduler extender 110 may be provided by one or more processors, including one or more of a graphics processor, an application processor, and another processor, where the one or more processors are located in the same place in a common semiconductor package.

[0034] The embodiments are not limited to any particular implementation or hosting of the XPU scheduler extender 110, and it is intended that one or more parts or components of the XPU scheduler extender 110 may be adopted or implemented as hardware, software, or firmware in any combination thereof. In one embodiment, for example, the XPU scheduler extender may be hosted by a machine learning processing unit different from the GPU. In another embodiment, the XPU scheduler extender may be distributed between the machine learning processing unit and the CPU. In yet another embodiment, the XPU scheduler extender may be distributed among the machine learning processing unit, the CPU, and the GPU. In yet another embodiment, the XPU scheduler extender may be distributed among the machine learning processing unit, the CPU, the GPU, and the hardware accelerator.

[0035] Furthermore, the embodiments are not limited to any particular implementation or hosting of the XPU scheduler extender 110, and one or more parts or components of the XPU scheduler extender 110 may be employed or implemented in more than one computing device (e.g., a host machine) 100, and are not intended to be limited to an implementation in a single computing device 100.

[0036] The computing device 100 can host network interface devices for providing access to networks such as LANs, wide area networks (WANs), metropolitan area networks (MANs), personal area networks (PANs), Bluetooth®, cloud networks, mobile networks (e.g., 3G, 4G, etc.), intranets, and the Internet. The network interface may include, for example, a wireless network interface having antennas that can represent one or more antennas. The network interface may also include, for example, a wired network interface for communicating with a remote device via a network cable, which may be, for example, an Ethernet® cable, a coaxial cable, a fiber optic cable, a serial cable, or a parallel cable.

[0037] The embodiments may be provided as a computer program product that includes one or more machine-readable media storing machine-executable instructions, which, when executed by one or more machines such as a computer, a computer network, or other electronic devices, can cause one or more machines to perform the operations according to the embodiments described herein. The machine-readable media may include, but are not limited to, floppy diskettes, optical disks, CD-ROMs (Compact Disk - Read-Only Memory), and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electronically Erasable Programmable Read-Only Memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0038] Furthermore, the embodiments may be downloaded as a computer program product, and the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by one or more data signals embodied and / or modulated on a carrier or other propagation medium via a communication link (e.g., a modem and / or network connection).

[0039] Throughout this document, the term "user" may be interchangeably referred to as "viewer," "observer," "speaker," "person," "individual," or "end-user." Throughout this document, terms such as "graphics domain" may be interchangeably referred to as "graphics processing unit," "graphics processor," or simply "GPU," and similarly, "CPU domain" or "host domain" may be interchangeably referred to as "computer processing unit," "application processor," or simply "CPU."

[0040] Please note that terms such as "node," "computing node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host machine," "device," "computing device," "computer," and "computing system" may be used interchangeably throughout this document. Furthermore, please note that terms such as "application," "software application," "program," "software program," "package," and "software package" may be used interchangeably throughout this document. In addition, terms such as "job," "input," "request," and "message" may be used interchangeably throughout this document.

[0041] Figure 2 illustrates the XPU scheduler extender 110 of Figure 1 in one implementation form of the present disclosure. For brevity, many of the details already described with reference to Figure 1 are not repeated or described below. In one embodiment, the XPU scheduler extender 110 may be the same as any of the XPU scheduler extenders 110, 120, 130, and 140 described with reference to Figure 1, and may include any number and types of components such as (without limitation) a metric collector 201, a metric analyzer 203, a resource allocation unit 205, and a security component 207. The embodiments are not limited to any particular implementation or hosting of the metric collector 201, metric analyzer 203, resource allocation unit 205, and security component 207. One or more parts or components of the metric collector 201, metric analyzer 203, resource allocation unit 205, and security component 207 may be employed or implemented in one or more computing devices (e.g., host machines) 100, and they are not intended to be limited to implementations in a single computing device 100. For example, the metric collector 201 and metric analyzer 203 may be hosted on multiple separate computing devices 100.

[0042] Computing device 100 is further shown to include a user interface 219 (e.g., a graphical user interface (GUI) based user interface, a web browser, a cloud-based platform user interface, a software application-based user interface, or other user or application programming interface (API)). Computing device 100 may further include an I / O source 104 having input components 231 such as a camera 242 (e.g., an Intel® RealSense® camera), sensors, and a microphone 241, and output components 233 such as a display device or simply a display 244 (e.g., an integrated display, a tensor display, a projection screen, a display screen, etc.), a speaker device or simply a speaker.

[0043] The computing device 100 is further described as accessing and / or communicating with one or more databases 225 and / or other computing devices via one or more communication media 230 (e.g., networks such as proximity networks, cloud networks, and the Internet).

[0044] In some embodiments, the database 225 may include one or more storage media or devices, repositories, data sources, etc., that have data and / or metadata of any quantity and type related to any number and type of applications, such as data and / or metadata related to one or more users, physical locations or areas, applicable laws, policies and / or regulations, user preferences and / or profiles, security and / or authentication data, history and / or other details.

[0045] As described above, the computing device 100 can host an I / O source 104 including an input component 231 and an output component 233. In one embodiment, the input component 231 may include a sensor array including, but not limited to, a microphone 241 (e.g., an ultrasonic microphone), a camera 242 (e.g., a two-dimensional (2D) camera, a three-dimensional (3D) camera, an infrared (IR) camera, a depth-sensing camera, etc.), a capacitor, a wireless component, a radar component, a scanner, and / or an accelerometer. Similarly, the output component 233 may include any number and type of display devices 244, a projector, light-emitting diodes (LEDs), a speaker 243, and / or a vibration motor, etc.

[0046] As previously mentioned, terms such as “logic,” “module,” “component,” “engine,” “circuit,” “element,” and “mechanism” can include, for example, software or hardware, and / or combinations thereof, such as firmware. For example, logic itself may be, include, or be associated with one or more devices, such as the XPU scheduler extender 120, XPU scheduler extender 130, and / or XPU scheduler extender 140, which are hosted by the application processor 112, graphics processor 116, and / or hardware accelerator 114 in Figure 1, respectively, and which need to facilitate or execute the corresponding logic to perform a particular task.

[0047] For example, as shown in the figure, the input component 231 may include any number and type of microphones 241, such as multiple microphones or a microphone array, such as ultrasonic microphones, dynamic microphones, fiber optic microphones, or laser microphones. One or more of the microphones 241 are intended to function as one or more input devices for receiving or receiving audio input (such as a human voice) to the computing device 100 and converting this audio or sound into an electrical signal. Similarly, one or more of the cameras 242 are intended to function as one or more input devices for detecting and capturing images and / or video of scenes, objects, etc., and providing the captured data to the computing device 100 as a video input.

[0048] As previously mentioned, conventional methods for scheduling scalable resources in microservice architectures have not been used, in particular, to deploy hardware accelerator resources in a distributed environment, especially to deploy XPU resources that may be distributed. Conventional methods also face latency and performance issues due to overutilization of hardware resources. Embodiments provide a novel technique for metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. This novel technique is used to address the aforementioned latency and / or performance issues in computing architectures that attempt to implement distributed XPU hardware accelerator resources in a microservice architecture. Implementations of this disclosure utilize an XPU scheduler extender 110 to provide metric and security-based accelerator service rescheduling and autoscaling using a programmable network device.

[0049] With respect to Figure 2, the XPU scheduler extender 110 includes a metric collector 201, a metric analyzer 203, a resource allocation unit 205, and a security component 207, which perform metric and security-based accelerator service rescheduling and autoscaling of the XPU scheduler extender 110. In the implementation of this disclosure, the operation of units 201, 203, 205, and 207 of the XPU scheduler extender 110 utilizes workload telemetry data, programmable network devices (e.g., IPUs), and XPU (heterogeneous processing unit) hardware accelerators (e.g., CPUs, GPUs, FPGAs, ASICs, inference accelerators, cryptographic accelerators, and other dedicated hardware accelerators) to achieve scalable and dynamic computing resource allocation to workloads that benefit from acceleration in computing clusters such as microservice architectures implemented in distributed computing environments.

[0050] The implementation provides an XPU scheduler extender 110 that communicates with a cluster node agent (not shown) that offers two usage models. In the first usage model, the XPU scheduler extender 110 communicates with the main cluster orchestrator (not shown) and the cluster node agent to relay information to the cluster regarding scaling the amount of XPU accelerator computing resources. In the second usage model, the XPU scheduler extender 110 communicates with the cluster node agent to configure more computing bandwidth for workloads that are already consuming XPU accelerator resources, and / or to switch the underlying XPU type for more efficient computing.

[0051] In the implementation described herein, the metric collector 201 and metric analyzer 203 of the XPU scheduler extender 110 collect and analyze metrics from service workloads. Metrics are telemetry data that can be mapped to functions that can be accelerated on XPU accelerator resources. For example, metrics such as the number of TLS handshakes per second, or the amount of bytes of data compressed on the CPU, can be mapped to encryption / compression accelerators by the resource allocation unit 205. Metric-based XPU scheduling can be used to make an optimized amount of XPU resources available to the cluster and allocate them to workloads that utilize them, while workloads that do not benefit from them can be scheduled to run on the CPU. The security component 207 of the XPU scheduler extender 110 operates to provide authentication and secure communication with any XPU accelerator resources scheduled to the cluster by the resource allocation unit 205.

[0052] Further details of the metric collector 201, metric analyzer 203, resource allocation unit 205, and security component 207 will be described later with reference to Figures 3 and 4.

[0053] Figure 3 shows a data center system 300 that provides metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, according to an implementation of the present disclosure. The data center system 300 shows an exemplary data center (e.g., hosted by a cloud service provider (CSP)) that provides various XPUs (heterogeneous processing units) for processing tasks in the data center, and the XPUs may include one or more of the following: a central processing unit (CPU) 315, graphics processing units (GPUs) 335 (including general-purpose GPUs (GPGPUs)), ASICs, or other processing units (e.g., accelerators 345, 355, 366, inference accelerator 345, cryptographic accelerator 355, programmable or fixed-function FPGA 364, application-specific integrated circuit (ASIC) 366, compression accelerator, etc.). The data center may also provide storage units for data storage tasks. The storage units may include, for example, solid-state drives (SSDs) 325. XPUs and / or storage units can be hosted in units of similar types (for example, CPU 315 hosted in application server 310, SSD 325 hosted in storage rack 320, GPU 335 hosted in GPU rack 330, inference accelerator 345 hosted in inference accelerator server 340, encryption accelerator 355 hosted in encryption accelerator rack 350, and general-purpose accelerators 362, 364, 366 hosted in accelerator rack 360).

[0054] The data center of System 300 provides various offloads to its host processing components 315, 325, 335, 345, 355, 362, 364, and 366, for example, by using IPUs 305 directly attached to each hosted processing component. While the IPU 305 is described for illustrative purposes, other programmable network devices such as DPUs or SmartNICs may be used interchangeably with the IPU 305 as described herein. The offloads provided may include networking, storage, security, etc. This allows the processing components 315, 325, 335, 345, 355, 362, 364, and 366 to run without a hypervisor, giving the CSP the ability to lease the entire data center host to security-conscious customers or to avoid crosstalk and other issues associated with multi-tenant hosts.

[0055] The IPU305 can play a critical role in data centers by providing data center operators such as cloud service providers (CSPs) and control points for security, acceleration, telemetry, and service orchestration. The IPU305 architecture can be built on existing smart network interface card (SmartNIC) functionality and is part of the control of security and data acceleration within and between distributed platforms. It is a secure domain controlled by the CSP to manage the platform, serve tenants, and ensure access to the data center network. The IPU305 enhances the performance and predictability of distributed runtimes and enables scaling to multi-terabit throughput by offloading host services, reliable transport, and optimizing data copy.

[0056] The IPU305 started as a basic NIC and has grown in complexity over the years, its sole purpose being to send and receive packets to and from the host. With the addition of networking software offload, the NIC has evolved into a SmartNIC capable of offloading functions such as VSwitch, VIRTIO-Net, and AVF. Remote distributed storage architectures offer a further evolution where compute and storage are no longer located in the same place, and large compute clusters are connected to large storage clusters over a network. This has been made possible by increased network speeds and advancements in protocols. One of the advantages that remote distributed storage offers beyond directly connected storage is the ability to develop and update compute and memory at different cadences. The amount of memory connected to a compute node is no longer limited by the physical addition or removal of hard drives, but can be hot-plugged as a PF to a PCIe switch. Technologies such as smart endpoints allow the IPU to have a firmware-controlled switch, along with the PCIe switch itself, so as not to be limited by the form of hardware implementation.

[0057] As described above, embodiments of this specification provide metric and security-based accelerator service rescheduling and autoscaling using programmable network devices. Embodiments provide a scheduler extender 317 (also referred to herein as the XPU scheduler extender) hosted by a CPU 315, running on an application server 310. In one implementation, the scheduler extender 317 is the same as the scheduler extender 110 described with respect to Figures 1 and 2. In one implementation, the scheduler extender 317 is provided to collect and analyze workload telemetry data (i.e., “metrics”) corresponding to communication links between microservices of services managed by a service mesh implemented in a container orchestration platform implemented in a data center system 300. Based on workload telemetry data, the scheduler extender 317 works with programmable network devices (such as the IPU 305) to scale up or down XPU hardware accelerators (e.g., CPU 315, GPU 335 and 362, FPGA 364, ASIC 366, inference accelerator 345, cryptographic accelerator 355, and / or other dedicated hardware accelerators) to achieve scalable and more dynamic allocation of computing resources to workloads that benefit from the acceleration of the computing cluster in the data center system 300.

[0058] The scheduler extender 317 and one or more node agents 307 (also referred to herein as cluster node agents 307) can provide multiple usage models for metric and security-based accelerator service rescheduling and autoscaling as described herein. In a first usage model, the scheduler extender 317 communicates with the main cluster orchestrator (not shown) and node agents 307 to relay information to the cluster regarding scaling the amount of XPU accelerator computing resources. In a second usage model, the scheduler extender 317 communicates with node agents 307 to configure more computing bandwidth for workloads that are already consuming XPU resources and / or switch the underlying XPU type for more efficient computing.

[0059] In the implementation described herein, the scheduler extender 317 collects and analyzes metrics from the workload. These metrics are telemetry data that can be mapped to functions that can also be accelerated on the XPU. For example, the number of TLS handshakes per second, or the amount of bytes of data compressed on the CPU, can be mapped to encryption / compression accelerators. Metric-based XPU scheduling can be used to make an optimized amount of XPU resources available to the cluster and allocate them to workloads that utilize them, while workloads that do not benefit from them can be scheduled to run on the CPU.

[0060] Figure 4 shows a block diagram of a data center system 400 that implements metric and security-based accelerator service rescheduling and autoscaling using programmable network devices, according to an implementation of the present disclosure. In one embodiment, the data center system 400 includes an application server / control plane 410, an XPU rack 420, and a data center server 440. In one implementation, the application server / control plane 410 may be the same as the application server 310 in Figure 3, and the XPU rack 420 may be, for example, the storage rack 320, GPU rack 330, inference accelerator rack 340, cryptographic accelerator rack 350, and / or general-purpose accelerator rack 360 in Figure 3. The data center server 440 may be a server computing device operating within the data center system 400 to provide management and orchestration functions for the data center system 400.

[0061] In one embodiment, the application server / control plane 410, XPU rack 420, and / or data center server 440 may be a computing device comprising a set of hardware, software, and firmware elements, and / or any combination of hardware, software, and / or firmware elements. For example, the application server / control plane 410, XPU rack 420, and / or data center server 440 may include hardware circuitry, such as one or more CPUs, GPUs, and hardware accelerators, to run one or more processes on the application server / control plane 410, XPU rack 420, and / or data center server 440, as described herein.

[0062] In some embodiments, the application server / control plane 410 includes a control plane scheduler 412, a node agent 414, and an XPU scheduler extender 416. In one embodiment, the control plane scheduler 412, the node agent 414, and / or the XPU scheduler extender 416 may be implemented on separate computing devices and connected in a communicative manner over a network (not shown). The control plane scheduler 412, the node agent 414, and / or the XPU scheduler extender 416 may be implemented using hardware circuitry such as one or more of a CPU, GPU, hardware accelerator, etc. In one embodiment, the control plane scheduler 412, the node agent 414, and / or the XPU scheduler extender 416 may be implemented using the computing system 100 described with respect to Figure 1. In one implementation, the XPU scheduler extender 416 is the same as the XPU scheduler extender 110 described with respect to Figure 1 and / or the scheduler extender 317 described with respect to Figure 3.

[0063] More generally, the exemplary control plane scheduler 412, node agent 414, and / or XPU scheduler extender 416 in Figure 4 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. For example, the exemplary control plane scheduler 412, node agent 414, and / or XPU scheduler extender 416 can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs).

[0064] In some embodiments, the XPU rack 420 includes an IPU cluster node agent 422 operating on an IPU 424 and an XPU resource 430 containing multiple XPUs 435. In one embodiment, the IPU cluster node agent 422, the IPU 424, and the XPU resource 430 can be implemented on separate computing devices and connected in a communicative manner via a network (not shown). The IPU cluster node agent 422, the IPU 424, and the XPU resource 430 can be implemented using hardware circuitry such as one or more of a CPU, GPU, hardware accelerator, etc. In one embodiment, the IPU cluster node agent 422, the IPU 424, and the XPU resource 430 can be implemented using the computing system 100 described with respect to Figure 1. In one implementation, the IPU cluster node agent 422 is the same as the node agent 307 described in Figure 3, the IPU 424 is the same as the IPU 305 described in Figure 3, and the XPU resources 430 and XPU 435 are the same as any of the CPU 315, GPU 335 and 362, FPGA 364, ASIC 366, inference accelerator 345, or cryptographic accelerator 355 described in Figure 3.

[0065] More generally, the exemplary IPU cluster node agent 422, IPU 424, and XPU resource 430 in Figure 4 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. For example, the IPU cluster node agent 422, IPU 424, and XPU resource 430 can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs).

[0066] In one embodiment, the data center server 440 includes a central resource orchestrator 445. The central resource orchestrator 445 may be implemented using hardware circuitry such as one or more of a CPU, GPU, hardware accelerator, etc. In one embodiment, the central resource orchestrator 445 may be implemented using the computing system 100 described with respect to Figure 1.

[0067] More generally, the exemplary central resource orchestrator 445 in Figure 4 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. For example, the central resource orchestrator 445 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs).

[0068] As described above, the data center system 400 implements metric and security-based accelerator service rescheduling and autoscaling using programmable network devices according to the implementation described herein. As shown in the figure, the data center system 400 may be implemented as a distributed computing environment using XPU hardware racks 420 (including IPUs 424) implemented separately from the application server / control plane 410 that runs the application workload. As shown in the figure, the control plane / scheduler 412, node agents 414, and applications (e.g., workloads 415) run on the application server (e.g., application server / control plane 410). An improved XPU computing balance can be maintained in the data center system 400 by bringing metric-based knowledge of application resource usage and enhanced XPU-aware scheduling capabilities provided by the XPU scheduler extender 416, as described herein.

[0069] One example of an embodiment described herein includes a containerized web application that may be implemented in a data center system 400. The web application consists of microservices, each running in its own container using the hardware resources of the data center system 400 (e.g., the CPUs of the application server / control plane 410 and / or the XPU 435 in the XPU rack 420), and communicating with other microservices using well-known protocols such as HTTP or gRPC. In one implementation, TLS can be used for the communication links between services.

[0070] Several tools, such as service meshes, enable granular collection of statistics for any given communication link between microservices. For example, a service mesh can report the number of new TLS connections to a given service or the number of bytes transferred per second. In one example, the metrics might show that there is a lot of traffic from the inventory service to the point-of-sale service, possibly as a result of repeated queries for inventory data (e.g., dynamic traffic patterns are common in modern web applications).

[0071] An implementation of this specification periodically rebalances the XPU resources 430 based on collected metrics (e.g., telemetry data) and known characteristics of the XPU accelerator devices (i.e., XPU 435) available in the data center. The XPU scheduler extender 416 may include a metric analyzer 417 for collecting and analyzing such metrics. The XPU scheduler extender 416 includes a resource allocation unit 418 to enable the dynamic scaling (e.g., adding or removing) of additional XPU resources 430 located on another platform and connected via a network, for the purpose of handling the workload 415 of services provided by the application server / control plane 410.

[0072] In an exemplary flow performed by the data center system 400, a particular microservice of a service is annotated by the service developer as one that can utilize a specific type of hardware accelerator. The metric analyzer 417 of the XPU scheduler extender 416 then collects metrics and analyzes relevant metrics related to the service in order to identify opportunities to provide acceleration to the service.

[0073] The metric analyzer 417 determines that the collected and analyzed metrics indicate potential improvements in workload processing (for example, improved query latency if compression is accelerated), and the resource allocation unit 418 of the XPU scheduler extender 416 provides the microservice service with another annotation (or a modified annotation) indicating that the microservice is a candidate to be scheduled on a particular type of XPU accelerator 435.

[0074] The resource allocation unit 418 also creates a scaling request to prepare and make available the XPU accelerator resource 435. In one implementation, the scaling request may be sent to the central resource orchestrator 445 to provision or update the XPU 435. In one implementation, the XPU scheduler extender 416 can communicate directly with the IPU 424 to have the XPU allocation unit 426 allocate or update the XPU 435 for the purpose of handling microservice workloads. If the XPU accelerator resource 435 is far from the application server / control plane 410, the scaling request may be sent to the IPU virtual node agent 422 to which the XPU 435 is connected.

[0075] In one implementation, the XPU allocation unit 426 within the IPU 424 can dynamically allocate the XPU 435, and the IPU 424 can register the XPU 435 with the application server / control plane 410. Depending on the rebalancing policy and scaling request, the microservices can wait for the next service restart (e.g., maintenance interruption) or can automatically restart with the dynamically added new XPU 435, for example.

[0076] Embodiments of this disclosure do not need to set any constraints on how rebalancing / provisioning policies are created and / or implemented. For example, a rebalancing / provisioning policy can be threshold-based. In such an example, a service can be marked as a candidate for acceleration if traffic exceeds a given threshold amount or if there are more than X TLS handshakes per second. However, rebalancing / provisioning policies can also be complex and based on technologies such as neural networks, and a rebalancing / provisioning policy can provide input as cluster-wide statistics. For example, it may inform about cluster utilization of hardware accelerator devices, expected future traffic patterns, etc. A rebalancing / provisioning policy can be updated as more information about workload properties becomes available. Dynamic and adaptive policies for rebalancing and / or scheduling can be used herein to improve system resource utilization in the implementations of this disclosure.

[0077] In one implementation, if the workload 415 is associated with security requirements and / or security properties of the XPU435 (e.g., the XPU435 should be FIP compliant or support SPDM), then all communication with the XPU435 should be protected. In one implementation, IPSec can be used as a solution to ensure that a new XPU435 added to a workload cluster can comply with the requirements. Enforcement of security policies can be performed at the IPU424, the control plane / scheduler 412, or both.

[0078] For example, regarding implementation on IPU424, the XPU scheduler extender 416 can communicate security requirements to IPU424 when submitting an XPU435 scaling request. The XPU allocation unit 426 can then make available an XPU435 that complies with the security requirements.

[0079] Regarding implementation in the control plane / scheduler 412, the XPU scheduler extender 416 can provide security policies to the control plane / scheduler 412. The IPU 424 can make the XPU 435 available, but before the XPU 435 is registered, the control plane / scheduler 412 must verify the attestation of the XPU 435 to ensure compliance with the security policy. A service (e.g., workload 415) can be accelerated and scheduled if it complies with the security policy. If there is a requirement for secure communication with the XPU 435, the control plane / scheduler 412 and / or the XPU scheduler extender 416 can establish a secure channel with the XPU before scheduling the acceleration of workload 415 via the XPU 435.

[0080] In one implementation, the XPU scheduler extender 416, which collects and analyzes metrics, can provide assurance that these metrics should not be used for malicious intent, such as side-channel attacks against the workload 415. This can be achieved by running the XPU scheduler extender 416 within a trusted execution environment (TEE), such as Intel® SGX®, AMD® SEV®, or Trustzone®. Communication between the worker node (e.g., the CPU of the application server / control plane 410) and the XPU 435 is also protected over the network using security technologies such as IPsec.

[0081] In some implementations, a cluster can also be used to run regular batch jobs that can benefit from XPU435 acceleration. These batch jobs can be completed regardless of which type of XPU435 is used for acceleration (e.g., compressed batch jobs implemented using FPGAs or dedicated ASICs). In this case, telemetry-based scheduling and / or scaling can be used to identify XPU435s for scheduling services based on, for example, historical performance history, environmental conditions (e.g., cluster power budget), or other service level agreements (SLAs). Another feature of the implementations herein is the inclusion of support for removing an XPU435 from a processing workload if another workload could benefit more from a particular XPU435 (e.g., to make a web application more powerful), while ensuring that application interruptions are limited.

[0082] Figure 5 is a flowchart illustrating an embodiment of Method 500 for metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. Method 500 can be implemented by processing logic which may include hardware (e.g., circuitry, dedicated logic, programmable logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. More specifically, Method 500 can be implemented in one or more modules as a set of logic instructions stored in mechanical or computer-readable storage media (also referred herein as non-temporary computer-readable storage media) such as RAM, ROM, PROM, firmware, flash memory, etc., configurable logic such as PLA, FPGA, CPLD, etc., fixed-function logic hardware using circuit technology such as ASIC, CMOS, or TTL technology, etc., or any combination thereof.

[0083] The process of Method 500 is shown in a linear sequence for brevity and clarity of presentation. However, any number of them can be executed in parallel, asynchronously, or in different orders. Furthermore, for brevity, clarity, and ease of understanding, many of the components and processes described with respect to Figures 1-4 may not be repeated or described below. In one implementation, a data center system implementing an XPU scheduler extender, such as the XPU scheduler extender 416 of the data center system 400 in Figure 4, can perform Method 500.

[0084] The exemplary process of Method 500 in Figure 5 begins in block 510, where a processing device running an XPU scheduler extender can collect metrics corresponding to communication links between microservices of a service managed by a service mesh. Next, in block 520, the processing device can determine, based on an analysis of the metrics by the XPU scheduler extender, that the service workload can be accelerated by offloading to an XPU hardware accelerator device (also referred to herein as a hardware accelerator device).

[0085] Next, in block 530, the processing device can generate a scaling request via the XPU scheduler extender to assign the XPU hardware accelerator device to a cluster of hardware devices configured for the service. In block 540, the processing device can have the XPU scheduler extender send the scaling request to a programmable network device that manages the XPU hardware accelerator device. In one implementation, the programmable network device assigns the XPU hardware accelerator device to the cluster and registers the XPU hardware accelerator device with the service mesh. Finally, in block 550, the processing device can schedule the service workload to the XPU hardware accelerator device.

[0086] Figure 6 is a flowchart illustrating an embodiment of Method 600 for security measures while implementing metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. Method 600 can be implemented by processing logic which may include hardware (e.g., circuits, dedicated logic, programmable logic, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. More specifically, Method 500 can be implemented in one or more modules as a set of logic instructions stored in mechanical or computer-readable storage media (also referred herein as non-temporary computer-readable storage media) such as RAM, ROM, PROM, firmware, flash memory, etc., configurable logic such as PLA, FPGA, CPLD, etc., fixed-function logic hardware using circuit technology such as ASIC, CMOS, or TTL technology, etc., or any combination thereof.

[0087] The process of Method 600 is shown in a linear sequence for brevity and clarity of presentation. However, any number of them can be executed in parallel, asynchronously, or in different orders. Furthermore, for brevity, clarity, and ease of understanding, many of the components and processes described with respect to Figures 1-5 may not be repeated or described below. In one implementation, a data center system implementing an XPU scheduler extender, such as the XPU scheduler extender 416 of the data center system 400 in Figure 4, can perform Method 600.

[0088] The exemplary process of Method 600 in Figure 6 begins in block 610, where the processing device can identify an XPU hardware accelerator device to include in a cluster of hardware devices configured for service in a service mesh. Next, in block 620, the processing device can verify the reliability of the XPU hardware accelerator device using an attestation protocol.

[0089] Next, in block 630, the processing device can establish a shared secret key with the XPU hardware accelerator device. Finally, in block 640, the processing device can communicate the shared secret key to the service mesh control plane scheduler for use when scheduling service workloads to the XPU hardware accelerator device.

[0090] Figure 7 is a schematic diagram of an exemplary electronic computing device 700 that enables metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, according to several embodiments. In some embodiments, the computing device 700 includes one or more processors 710, including one or more processor cores 718, which include one or more XPU scheduler extenders (XSEs) 715, such as the XPU scheduler extenders 110-140 described with respect to Figures 1 and 2, the scheduler extender 317 described with respect to Figure 3, or the XPU scheduler extender 416 described with respect to Figure 4. In some embodiments, the computing device 700 includes a hardware accelerator 768, which includes an XPU scheduler extender 782, such as the XPU scheduler extenders 110-140 described with respect to Figures 1 and 2, the scheduler extender 317 described with respect to Figure 3, or the XPU scheduler extender 416 described with respect to Figure 4. In some embodiments, the computing device uses a programmable network device, as shown in Figures 1 to 6, to provide low-metric and security-based accelerator service rescheduling and autoscaling.

[0091] The computing device 700 may further include one or more of the following: a cache 762, a graphical processing unit (GPU) 712 (which may be a hardware accelerator in some implementations), a wireless input / output (I / O) interface 720, a wired I / O interface 730, system memory 740 (e.g., memory circuitry), power management circuitry 750, a non-temporary storage device 760, and a network interface 770 for connecting to a network 772. The following description provides a brief and general description of the components that make up an exemplary computing device 700. As an example, an unspecified computing device 700 may include a desktop computing device, a blade server device, a workstation, or a similar device or system.

[0092] In the embodiment, the processor core 718 can execute a machine-readable instruction set 714, read data and / or the instruction set 714 from one or more storage devices 760, and write data to one or more storage devices 760. Those skilled in the art will understand that the illustrated embodiment and other embodiments can be implemented in other processor-based device configurations, including, for example, portable electronic or handheld electronic devices such as smartphones, portable computers, wearable computers, home appliances, personal computers ("PCs"), network PCs, minicomputers, server blades, and mainframe computers.

[0093] The processor core 718 may include any number of hardwired or configurable circuits, some or all of which may include programmable and / or configurable combinations of electronic components, semiconductor devices, and / or logic elements, which are partially or entirely located in a PC, server, or other computing system capable of executing processor-readable instructions.

[0094] The computing device 700 includes a bus or similar communication link 716 that communicatively connects and facilitates the exchange of information and / or data between various system components, including a processor core 718, a cache 762, a graphics processor circuit 712, one or more wireless I / O interfaces 720, one or more wired I / O interfaces 730, one or more storage devices 760, and / or one or more network interfaces 770. Although the computing device 700 may be referred to in the singular form herein, this is not intended to limit embodiments to a single computing device 700, because in particular embodiments there may be more than one computing device 700 that incorporate, include, or house any number of communicably connected, collocated, or remotely networked circuits or devices.

[0095] The 718 processor core may include any number, type, or combination of currently available or future-developed devices capable of executing machine-readable instruction sets.

[0096] The processor core 718 may include (or be coupled to) any currently or future-developed single-core or multi-core processor or microprocessor, such as one or more system-on-a-chip (SOCs), central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable logic units, field-programmable gate arrays (FPGAs), etc. Unless otherwise specified, the configuration and operation of the various blocks shown in Figure 7 are of conventional designs. Therefore, such blocks should be understood by those skilled in the art and are not described in further detail herein. The bus 716 interconnecting at least some of the components of the computing device 700 may employ any currently available or future-developed serial or parallel bus structure or architecture.

[0097] The system memory 740 may include read-only memory ("ROM") 742 and random access memory ("RAM") 746. A portion of the ROM 742 can be used to store or hold a basic input / output system ("BIOS") 744. The BIOS 744 provides basic functionality to the computing device 700 by, for example, causing a processor core 718 to load and / or execute one or more machine-readable instruction sets 714. In embodiments, at least some of the one or more machine-readable instruction sets 714 cause at least a portion of the processor core 718 to provide, create, generate, transfer, and / or function a dedicated, special, specific machine, such as a word processing machine, a digital image acquisition machine, a media playing machine, a game system, a communication device, a smartphone, and the like.

[0098] The computing device 700 may include at least one wireless input / output (I / O) interface 720. At least one wireless I / O interface 720 may be communicatively connected to one or more physical output devices 722 (such as a haptic device, video display, audio output device, or hardcopy output device). At least one wireless I / O interface 720 may be communicatively connected to one or more physical input devices 724 (such as a pointing device, touchscreen, keyboard, or haptic device). At least one wireless I / O interface 720 may include any currently available or future-developed wireless I / O interface. Exemplary wireless I / O interfaces include, but are not limited to, Bluetooth® and Near Field Communication (NFC).

[0099] The computing device 700 may include one or more wired input / output (I / O) interfaces 730. At least one wired I / O interface 730 may be communicatively connected to one or more physical output devices 722 (such as a haptic device, video display, audio output device, hard copy output device, etc.). At least one wired I / O interface 730 may be communicatively connected to one or more physical input devices 724 (such as a pointing device, touchscreen, keyboard, haptic device, etc.). The wired I / O interface 730 may include any I / O interface currently available or to be developed in the future. Exemplary wired I / O interfaces include, but are not limited to, Universal Serial Bus (USB) and IEEE 1394 ("FireWire®").

[0100] The computing device 700 may include one or more non-temporary data storage devices 760 that are communicatively connected. The data storage devices 760 may include one or more hard disk drives (HDDs) and / or one or more solid-state storage devices (SSDs). The one or more data storage devices 760 may include any currently or future-developed storage equipment, network storage devices, and / or systems. Non-limiting examples of such data storage devices 760 include, but are not limited to, any currently or future-developed non-temporary storage equipment or devices, such as one or more magnetic storage devices, one or more optical storage devices, one or more electroresistive storage devices, one or more molecular storage devices, one or more quantum storage devices, or various combinations thereof. In some implementations, the one or more data storage devices 760 may include one or more removable storage devices, such as one or more flash drives, flash memory, flash storage units, or similar equipment or devices that can communicatively connect to and disconnect from the computing device 700.

[0101] One or more data storage devices 760 may include an interface or controller (not shown) that communicatively connects each storage device or system to the bus 716. One or more data storage devices 760 may store, hold, or store machine-readable instruction sets, data structures, program modules, data stores, databases, logical structures, and / or other data useful for the processor core 718 and / or graphics processor circuit 712 and / or one or more applications running on or therein. In some cases, one or more data storage devices 760 may be communicatively connected to the processor core 718, one or more wireless communication interfaces 720 (e.g., Bluetooth®, Near Field Communication or NFC), and / or one or more network interfaces 770 (such as IEEE 802.3 or Ethernet®, IEEE 802.11, or Wi-Fi®) for, for example, via the bus 716 or via one or more wired communication interfaces 730 (e.g., Universal Serial Bus or USB).

[0102] Processor-readable instruction sets 714 and other programs, applications, logic sets, and / or modules can be stored in system memory 740, either entirely or partially. Such instruction sets 714 may be transferred, either entirely or partially, from one or more data storage devices 760. Instruction sets 714 may be loaded, stored, or held in system memory 740, either entirely or partially, while being executed by the processor core 718 and / or graphics processor circuitry 712.

[0103] The computing device 700 may include a power management circuit 750 that controls one or more operating modes of the energy storage device 752. In embodiments, the energy storage device 752 may include one or more primary (i.e., non-rechargeable) or secondary (i.e., rechargeable) batteries or similar energy storage devices. In embodiments, the energy storage device 752 may include one or more supercapacitors or ultracapacitors. In embodiments, the power management circuit 750 may change, adjust, or control the flow of energy from the external power source 754 to the energy storage device 752 and / or the computing device 700. The power source 754 may include, but is not limited to, a photovoltaic system, a commercial power grid, a portable generator, an external energy storage device, or any combination thereof.

[0104] For convenience, the processor core 718, graphics processor circuit 712, wireless I / O interface 720, wired I / O interface 730, storage device 760, and network interface 770 are shown as being communicatively connected to each other via bus 716, thereby providing connectivity between the components described above. In alternative embodiments, the components described above may be communicatively connected in a manner different from that shown in Figure 7. For example, one or more of the components described above may be directly connected to other components, or connected to each other via one or more intermediate components (not shown). In another example, one or more of the components described above may be integrated into the processor core 718 and / or graphics processor circuit 712. In some embodiments, all or part of bus 716 may be omitted, and the components may be directly connected to each other using appropriate wired or wireless connectivity.

[0105] The following examples relate to further embodiments. Example 1 is a device that facilitates metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. The device of Example 1 comprises one or more processors for collecting metrics corresponding to communication links between microservices of a service managed by a service mesh, determining, based on an analysis of the metrics, that the service workload can be accelerated by offloading to a hardware accelerator device, generating a scaling request for assigning the hardware accelerator device to a cluster of hardware devices configured for the service, sending the scaling request to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster, registering the hardware accelerator device with the service mesh, and scheduling the service workload to the hardware accelerator device.

[0106] In Example 2, the subject of Example 1 optionally includes telemetry data, which includes at least one of the following: the number of new Transport Layer Security (TLS) connections, the number of bytes transferred per second, the traffic patterns between microservices, or the utilization of hardware devices in the cluster.

[0107] In Example 3, any one subject from Examples 1 to 2 may optionally include one or more processors that identify that a workload can be accelerated, further comprising one or more processors that annotate a service to indicate that the service is a candidate for scheduling on a hardware accelerator device, the annotation causing a service mesh control plane scheduler to schedule the service to a hardware accelerator device. In Example 4, any one subject from Examples 1 to 3 may optionally include one or more processors that determine, based on an analysis of metrics, that a workload can be accelerated by offloading it to a determined type of hardware accelerator device, which includes at least one of a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a cryptographic accelerator device, an inference accelerator device, or a compression accelerator device.

[0108] In Example 5, any one of the themes from Examples 1 to 4 may optionally include a scaling request being communicated to a central resource orchestrator in a data center hosting one or more processors and hardware accelerator devices, the central resource orchestrator managing a set of data center hardware resources hosting at least one or more processors, programmable network devices, and hardware accelerator devices.

[0109] In Example 6, any one subject from Examples 1 to 5 may optionally include one or more processors comprising a scheduler extender circuit for extending the operation of a service mesh control plane scheduler, the control plane scheduler scheduling service workloads on one or more available hardware resources in a data center, and one or more available hardware resources comprising at least a hardware accelerator device. In Example 7, any one subject from Examples 1 to 6 may optionally include one or more processors further authenticating hardware accelerator devices as part of assigning the hardware accelerator devices to a cluster.

[0110] In Example 8, the subject of any one of Examples 1 through 7 may optionally include one or more processors establishing a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device. In Example 9, the subject of any one of Examples 1 through 8 may optionally include one or more processors running a scheduler extender within a trusted execution environment (TEE) to isolate the scheduler extender, which performs tasks such as collecting, determining, generating, and causing.

[0111] In Example 10, any one subject from Examples 1 through 9 may optionally include one or more processors identifying a hardware accelerator based on the hardware accelerator's past performance history, the hardware accelerator's environmental conditions, or a service level agreement (SLA) corresponding to the hardware accelerator's services. In Example 11, any one subject from Examples 1 through 10 may optionally include one or more processors further communicating with a cluster node agent running on a programmable network device, the cluster node agent configuring bandwidth on one or more hardware resources in the cluster based on scaling requests.

[0112] In Example 12, any one subject from Examples 1 to 11 may optionally include the programmable network device and hardware accelerator device being distributed from one or more processors, the programmable network device, and one or more processors in a separate server device in a data center hosting the hardware accelerator device. In Example 13, any one subject from Examples 1 to 12 may optionally include the programmable network device comprising at least one of an infrastructure processing unit (IPU) or a data processing unit (DPU).

[0113] Example 14 is a non-temporary computer-readable storage medium for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. The non-temporary computer-readable storage medium of Example 14 stores executable computer program instructions that cause one or more processors to perform an operation when executed by one or more processors, the operation comprising: collecting metrics corresponding to communication links between microservices of a service managed by a service mesh by one or more processors; determining, based on an analysis of the metrics, that the service workload can be accelerated by offloading to a hardware accelerator device; generating a scaling request for assigning the hardware accelerator device to a cluster of hardware devices configured for the service; sending the scaling request to a programmable network device managing the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster and register the hardware accelerator device with the service mesh; and scheduling the service workload to the hardware accelerator device.

[0114] In Example 15, the subject matter of Example 14 may optionally include a scaling request being communicated to a central resource orchestrator in a data center hosting one or more processors and hardware accelerator devices, the central resource orchestrator managing a set of hardware resources in the data center hosting at least one or more processors, programmable network devices, and hardware accelerator devices. In Example 16, the subject matter of Examples 14 to 15 may optionally include an operation further including authenticating hardware accelerator devices as part of assigning hardware accelerator devices to a cluster.

[0115] In Example 17, the subject matter of Examples 14 through 16 may optionally further include the operation establishing a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device.

[0116] Example 18 is a method for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. The method in Example 18 may include: one or more processors collecting metrics corresponding to communication links between microservices of a service managed by a service mesh; one or more processors analyzing the metrics to determine whether the service workload can be accelerated by offloading to a hardware accelerator device; one or more processors generating a scaling request to assign the hardware accelerator device to a cluster of hardware devices configured for the service; one or more processors sending the scaling request to a programmable network device managing the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster and register the hardware accelerator device with the service mesh; and one or more processors scheduling the service workload to the hardware accelerator device.

[0117] In Example 19, the subject of Example 18 may optionally include a scaling request being communicated to a central resource orchestrator in a data center hosting one or more processors and hardware accelerator devices, the central resource orchestrator managing a set of data center hardware resources hosting at least one or more processors, programmable network devices, and hardware accelerator devices. In Example 20, the subject of Examples 18 to 19 may optionally further include communicating with a cluster node agent running on a programmable network device, the cluster node agent configuring bandwidth with one or more hardware resources in the cluster based on the scaling request.

[0118] Example 21 is a system for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device. The system in Example 21 may optionally include memory for storing blocks of data and a processor communicatively coupled to the memory to collect metrics corresponding to communication links between microservices of a service managed by a service mesh, the processor collects metrics corresponding to communication links between microservices of a service managed by a service mesh, determines, based on an analysis of the metrics, that the service workload can be accelerated by offloading to a hardware accelerator device, generates a scaling request to assign the hardware accelerator device to a cluster of hardware devices configured for the service, sends the scaling request to a programmable network device that manages the hardware accelerator device, causes the programmable network device to assign the hardware accelerator device to the cluster, registers the hardware accelerator device with the service mesh, and schedules the service workload to the hardware accelerator device.

[0119] In Example 22, the subject of Example 21 optionally includes that the metrics include telemetry data, which includes at least one of the following: the number of new Transport Layer Security (TLS) connections, the number of bytes transferred per second, the traffic patterns between microservices, or the utilization of hardware devices in the cluster.

[0120] In Example 23, any one subject from Examples 21 to 22 may optionally include one or more processors that identify that a workload can be accelerated further comprising one or more processors that annotate a service to indicate that the service is a candidate for scheduling on a hardware accelerator device, the annotation causing a service mesh control plane scheduler to schedule the service to a hardware accelerator device. In Example 24, any one subject from Examples 21 to 23 may optionally include one or more processors that determine, based on an analysis of metrics, that a workload can be accelerated by offloading it to a determined type of hardware accelerator device, which includes at least one of a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a cryptographic accelerator device, an inference accelerator device, or a compression accelerator device.

[0121] In Example 25, any one subject from Examples 21 to 24 may optionally include a scaling request being communicated to a central resource orchestrator in a data center hosting one or more processors and hardware accelerator devices, the central resource orchestrator managing a set of hardware resources in the data center hosting at least one or more processors, programmable network devices, and hardware accelerator devices.

[0122] In Example 26, any one subject from Examples 21 to 25 may optionally include one or more processors further extending the operation of a service mesh control plane scheduler, the control plane scheduler scheduling service workloads on one or more available hardware resources in a data center, and one or more available hardware resources comprising at least a hardware accelerator device. In Example 27, any one subject from Examples 21 to 26 may optionally include one or more processors further authenticating a hardware accelerator device as part of assigning the hardware accelerator device to a cluster.

[0123] In Example 28, any one subject from Examples 21 to 27 may optionally include one or more processors establishing a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device. In Example 29, any one subject from Examples 21 to 28 may optionally include one or more processors running a scheduler extender within a trusted execution environment (TEE) to isolate the scheduler extender, which performs tasks such as collecting, determining, generating, and causing.

[0124] In Example 30, any one subject from Examples 21 to 29 may optionally include one or more processors identifying a hardware accelerator based on the hardware accelerator's past performance history, the hardware accelerator's environmental conditions, or a service level agreement (SLA) corresponding to the hardware accelerator's services. In Example 31, any one subject from Examples 21 to 30 may optionally include one or more processors further communicating with a cluster node agent running on a programmable network device, the cluster node agent configuring bandwidth on one or more hardware resources in the cluster based on scaling requests.

[0125] In Example 32, any one subject from Examples 21 to 31 may optionally include the programmable network device and hardware accelerator device being distributed from one or more processors, one or more processors in a separate server device in a data center hosting the programmable network device and hardware accelerator device. In Example 33, any one subject from Examples 21 to 32 may optionally include the programmable network device comprising at least one of an infrastructure processing unit (IPU) or a data processing unit (DPU).

[0126] Example 34 is a device for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, the device comprising means for collecting metrics corresponding to communication links between microservices of a service managed by a service mesh; means for determining, based on an analysis of the metrics, that the service workload can be accelerated by offloading to a hardware accelerator device; means for generating a scaling request to assign the hardware accelerator device to a cluster of hardware devices configured for the service; means for causing the scaling request to be sent to a programmable network device managing the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster and register the hardware accelerator device with the service mesh; and means for scheduling the service workload to the hardware accelerator device. In Example 35, the subject of Example 34 may optionally include a device further configured to perform any one of the methods of Examples 19 to 20.

[0127] Example 36 is at least one machine-readable medium containing a plurality of instructions that cause a computing device to perform the method described in any one of Examples 18 to 20 in response to being executed on the computing device. Example 37 is a device for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device configured to perform the method described in any one of Examples 18 to 20. Example 38 is a device for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, comprising means for performing the method described in any one of items 18 to 20. The specifications in the examples can be used in any one or more embodiments.

[0128] The foregoing description and drawings should be considered illustrative rather than restrictive. Those skilled in the art will understand that various modifications and changes can be made to the embodiments described herein without departing from the broader spirit and scope of the features set forth in the appended claims. [Other possible items] [Item 1] It is a device, We collect metrics corresponding to the communication links between microservices of a service managed by a service mesh. Based on the analysis of the aforementioned metrics, it is determined that the workload of the service can be accelerated by offloading it to a hardware accelerator device. The hardware accelerator device generates a scaling request to be allocated to a cluster of hardware devices configured for the service. The scaling request is sent to the programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster and register the hardware accelerator device with the service mesh. The workload of the service is scheduled to the hardware accelerator device. A device comprising one or more processors for the purpose of [Item 2] The apparatus as described in item 1, wherein the metric includes telemetry data including at least one of the following: the number of new transport layer security (TLS) connections, the number of bytes transferred per second, the traffic patterns between the microservices, or the utilization rate of the cluster's hardware devices. [Item 3] The apparatus according to item 1, wherein one or more processors that identify the ability to accelerate the workload further annotate the service to indicate that the service is a candidate for scheduling in the hardware accelerator device, and the annotation causes the control plane scheduler of the service mesh to schedule the service to the hardware accelerator device. [Item 4] The apparatus according to item 1, wherein one or more processors determine, based on the analysis of the metric, that they can accelerate the workload by offloading it to a determined type of hardware accelerator device, which includes at least one of a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a cryptographic accelerator device, an inference accelerator device, or a compression accelerator device. [Item 5] The apparatus according to item 1, wherein the scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device, and the hardware accelerator device. [Item 6] The apparatus according to item 1, wherein the one or more processors include a scheduler extender circuit for extending the operation of the service mesh control plane scheduler, the control plane scheduler schedules the workload of the service on one or more available hardware resources of the data center, and the one or more available hardware resources include at least the hardware accelerator device. [Item 7] The apparatus according to item 1, wherein one or more processors further authenticate the hardware accelerator device as part of assigning the hardware accelerator device to the cluster. [Item 8] The apparatus according to item 7, wherein the one or more processors further establish a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device. [Item 9] The apparatus according to item 1, wherein one or more processors execute a scheduler extender within a trusted execution environment (TEE) to isolate the scheduler extender, and the scheduler extender performs the tasks of collecting, determining, generating, and causing. [Item 10] The apparatus according to item 1, wherein one or more processors identify the hardware accelerator based on the hardware accelerator's past performance history, the hardware accelerator's environmental conditions, or a service level agreement (SLA) corresponding to the hardware accelerator's services. [Item 11] The apparatus according to item 1, wherein the one or more processors further communicate with cluster node agents running on the programmable network device, and the cluster node agents configure bandwidth using one or more hardware resources of the cluster based on the scaling request. [Item 12] The apparatus according to item 1, wherein the programmable network device and the hardware accelerator device are distributed from the one or more processors, the programmable network device, and the one or more processors in a separate server device in a data center hosting the hardware accelerator device. [Item 13] The programmable network device is the apparatus described in item 1, comprising at least one of an infrastructure processing unit (IPU) or a data processing unit (DPU). [Item 14] A non-temporary computer-readable storage medium storing executable computer program instructions that cause one or more processors to perform an operation, wherein the operation is: Collecting metrics corresponding to communication links between microservices of a service managed by a service mesh using one or more processors, Based on the analysis of the aforementioned metrics, it is determined that the workload of the service can be accelerated by offloading it to a hardware accelerator device. The hardware accelerator device generates a scaling request to be allocated to a cluster of hardware devices configured for the service, The scaling request is sent to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster and register the hardware accelerator device with the service mesh. A non-temporary computer-readable storage medium, which includes scheduling the workload of the service to the hardware accelerator device. [Item 15] The scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device, and the hardware accelerator device, as described in item 14. [Item 16] The non-temporary computer-readable storage medium described in item 14 further comprises authenticating the hardware accelerator device as part of assigning the hardware accelerator device to the cluster. [Item 17] The non-temporary computer-readable storage medium described in item 16 further comprises establishing a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device. [Item 18] It is a method, The process involves one or more processors collecting metrics corresponding to the communication links between microservices of a service managed by a service mesh, The steps include determining, based on an analysis of the metric by one or more processors, that the workload of the service can be accelerated by offloading to a hardware accelerator device, The steps include: generating a scaling request by one or more processors to allocate the hardware accelerator device to a cluster of hardware devices configured for the service; The steps include: causing one or more processors to send the scaling request to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster, and registering the hardware accelerator device with the service mesh; A method comprising the step of scheduling the workload of the service to the hardware accelerator device using one or more processors. [Item 19] The method according to item 18, wherein the scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device, and the hardware accelerator device. [Item 20] The method according to item 18, wherein the scheduler extender communicates with a cluster node agent running on the programmable network device, and the cluster node agent configures bandwidth on one or more hardware resources of the cluster based on the scaling request.

Claims

1. It is a device, We collect metrics corresponding to the communication links between microservices of a service managed by a service mesh. Based on the analysis of the aforementioned metrics, it is determined that the workload of the service can be accelerated by offloading it to a hardware accelerator device. The hardware accelerator device generates a scaling request to be allocated to a cluster of hardware devices configured for the service. The scaling request is sent to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster, and causing the programmable network device to register the hardware accelerator device with the service mesh. The workload of the service is scheduled to the hardware accelerator device. A device comprising one or more processors for the purpose of

2. The apparatus according to claim 1, wherein the metric includes telemetry data comprising at least one of the following: the number of new transport layer security (TLS) connections, the number of bytes transferred per second, the traffic patterns between the microservices, or the utilization rate of the hardware devices of the cluster.

3. The apparatus according to claim 1, wherein one or more processors that identify the workload can be accelerated further annotate the service to indicate that the service is a candidate for scheduling in the hardware accelerator device, the annotation causes the control plane scheduler of the service mesh to schedule the service to the hardware accelerator device.

4. The apparatus according to claim 1, wherein one or more processors determine, based on the analysis of the metric, that they can accelerate the workload by offloading it to a determined type of hardware accelerator device, which includes at least one of a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a cryptographic accelerator device, an inference accelerator device, or a compression accelerator device.

5. The apparatus according to claim 1, wherein the scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device and the hardware accelerator device.

6. The apparatus according to claim 1, wherein the one or more processors have a scheduler extender circuit for extending the operation of the control plane scheduler of the service mesh, the control plane scheduler schedules the workload of the service on one or more available hardware resources of the data center, and the one or more available hardware resources have at least the hardware accelerator device.

7. The apparatus according to claim 1, wherein the one or more processors further authenticate the hardware accelerator device as part of assigning the hardware accelerator device to the cluster.

8. The apparatus according to claim 1, wherein the one or more processors further establish a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler for scheduling workloads to the hardware accelerator device.

9. The apparatus according to claim 1, wherein one or more processors execute a scheduler extender in a trusted execution environment (TEE) and isolate the scheduler extender, and the scheduler extender performs the collecting, determining, generating, and generating.

10. The apparatus according to claim 1, wherein one or more processors identify the hardware accelerator device based on the past performance history of the hardware accelerator device, the environmental conditions of the hardware accelerator device, or a service level agreement (SLA) corresponding to the service of the hardware accelerator device.

11. The apparatus according to claim 1, wherein the one or more processors further communicate with a cluster node agent running on the programmable network device, and the cluster node agent configures bandwidth on one or more hardware resources of the cluster based on the scaling request.

12. The apparatus according to claim 1, wherein the programmable network device and the hardware accelerator device are distributed from the one or more processors of a separate server device in a data center hosting the one or more processors, the programmable network device, and the hardware accelerator device.

13. The apparatus according to any one of claims 1 to 12, wherein the programmable network device comprises at least one of an infrastructure processing unit (IPU) or a data processing unit (DPU).

14. One or more processors, A procedure for collecting metrics corresponding to communication links between microservices of a service managed by a service mesh using one or more processors, A procedure for determining, based on an analysis of the aforementioned metrics, that the workload of the service can be accelerated by offloading it to a hardware accelerator device, A procedure for generating a scaling request to allocate the hardware accelerator device to a cluster of hardware devices configured for the service, A procedure to cause the scaling request to be sent to a programmable network device that manages the hardware accelerator device, to cause the programmable network device to assign the hardware accelerator device to the cluster, and to cause the programmable network device to register the hardware accelerator device with the service mesh, A computer program for causing the hardware accelerator device to perform a procedure for scheduling the workload of the service.

15. The computer program according to claim 14, wherein the scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device, and the hardware accelerator device.

16. The computer program according to claim 14, wherein the procedure further includes authenticating the hardware accelerator device as part of assigning the hardware accelerator device to the cluster.

17. The computer program according to claim 14, further comprising establishing a shared secret key for use by the hardware accelerator device for secure communication between the hardware accelerator device and the service mesh control plane scheduler that schedules workloads to the hardware accelerator device.

18. It is a method, The steps include: collecting metrics corresponding to communication links between microservices of a service managed by a service mesh using one or more processors; The steps include determining, based on an analysis of the metric by one or more processors, that the workload of the service can be accelerated by offloading to a hardware accelerator device, The steps include: generating a scaling request by one or more processors to allocate the hardware accelerator device to a cluster of hardware devices configured for the service; The steps include: causing one or more processors to send the scaling request to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster, and causing the programmable network device to register the hardware accelerator device with the service mesh; The step of scheduling the workload of the service to the hardware accelerator device using one or more processors. A method that includes [a certain feature].

19. The method according to claim 18, wherein the scaling request is communicated to a central resource orchestrator in a data center hosting the one or more processors and the hardware accelerator device, and the central resource orchestrator manages a set of hardware resources in the data center hosting at least the one or more processors, the programmable network device and the hardware accelerator device.

20. The method according to claim 18, wherein the scheduler extender communicates with a cluster node agent running on the programmable network device, and the cluster node agent configures bandwidth using one or more hardware resources of the cluster based on the scaling request.

21. A system for facilitating metric and security-based accelerator service rescheduling and auto-scaling using programmable network devices, Memory for storing blocks of data, A processor connected to the aforementioned memory in a communicative manner The processor includes, We collect metrics corresponding to the communication links between microservices of a service managed by a service mesh. Based on the analysis of the aforementioned metrics, it is determined that the workload of the service can be accelerated by offloading it to a hardware accelerator device. The hardware accelerator device generates a scaling request to be allocated to a cluster of hardware devices configured for the service. The scaling request is sent to a programmable network device that manages the hardware accelerator device, causing the programmable network device to assign the hardware accelerator device to the cluster, and causing the programmable network device to register the hardware accelerator device with the service mesh. A system for scheduling the workload of the service to the hardware accelerator device.

22. The system according to claim 21, wherein the metric includes telemetry data comprising at least one of the following: the number of new transport layer security (TLS) connections, the number of bytes transferred per second, the traffic patterns between the microservices, or the utilization rate of the cluster's hardware devices.

23. The system according to claim 21, wherein the processor that identifies the workload can be accelerated further comprises the processor annotating the service to indicate that the service is a candidate for scheduling in the hardware accelerator device, the annotation causing the control plane scheduler of the service mesh to schedule the service to the hardware accelerator device.

24. The system according to any one of claims 21 to 23, wherein the processor determines, based on the analysis of the metric, that it can accelerate the workload by offloading it to a determined type of hardware accelerator device, which includes at least one of a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a cryptographic accelerator device, an inference accelerator device, or a compression accelerator device.

25. An apparatus for facilitating metric and security-based accelerator service rescheduling and autoscaling using a programmable network device, comprising means for performing the method according to any one of claims 18 to 20.

26. A non-temporary computer-readable storage medium storing a computer program according to any one of claims 14 to 17.

Images