Method, electronic system and electronic apparatus for allocating machine learning task to computing device

Abstract

Methods, electronic systems, and electronic apparatuses for assigning machine learning tasks to computing devices are disclosed. The method may include: determining characteristics of a machine learning task; determining a characteristic of a computing system, where the computing system may include one or more computing devices; and assigning the machine learning task to at least one of the one or more computing devices based on the characteristics of the machine learning task and the characteristics of the computing system. The characteristics of the machine learning tasks may include at least one of compatibility, priority, order, size, and type. The characteristics of the machine learning task may include at least one of performance compatibility, efficiency compatibility, and latency compatibility. The characteristics of the computing system may include at least one of a policy, a topology, a state, an operating parameter, and a scheduling algorithm. The characteristics of the computing system may include at least one of a performance policy and an efficiency policy.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 530,471 filed on August 2, 2023 and U.S. Patent Application No. 18 / 780,430 filed on July 22, 2024, which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to computing devices, and more particularly to methods, electronic systems, and electronic devices for assigning machine learning tasks to computing devices. Background Art

[0003] A computing system may use one or more computing devices to improve the performance of the system. Some computing workloads (such as machine learning workloads) may involve the use of models that may be run on one or more computing devices. The models may be used, for example, to perform one or more machine learning tasks (such as inference).

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not constitute prior art. Summary of the invention

[0005] A method may include: determining, by at least one processor, a characteristic of a machine learning task; determining, by the at least one processor, a characteristic of a computing system, wherein the computing system may include one or more computing devices; and assigning, by the at least one processor, the machine learning task to at least one of the one or more computing devices based on the characteristic of the machine learning task and the characteristic of the computing system. The characteristic of the machine learning task may include at least one of compatibility, priority, order, size, or type. The characteristic of the machine learning task may include at least one of performance compatibility, efficiency compatibility, or latency compatibility. The characteristic of the computing system may include at least one of a policy, a topology, a state, an operating parameter, or a scheduling algorithm. The characteristic of the computing system may include at least one of a performance policy or an efficiency policy. The characteristic of the computing system may include a policy, the characteristic of the machine learning task may include: a first compatibility with a first computing device in one or more computing devices based on the policy, and a second compatibility with a second computing device in one or more computing devices based on the policy, and the step of assigning may include: assigning the machine learning task to the first computing device in the one or more computing devices based on the policy and the first compatibility. The characteristic of the computing system may include a first policy and a second policy, the characteristic of the machine learning task may include a first compatibility with a first computing device in one or more computing devices based on the first policy, and a second compatibility with a second computing device in one or more computing devices based on the second policy, and the step of allocating may include: allocating the machine learning task to the first computing device in the one or more computing devices based on the first policy and the first compatibility. The machine learning task is a first machine learning task, the characteristic of the machine learning task is a first characteristic of the first machine learning task, and the step of allocating may include: selecting the first machine learning task based on the first characteristic of the first machine learning task, the second characteristic of the second machine learning task, and a scheduling algorithm, and allocating the first machine learning task to at least one of the one or more computing devices based on the step of selecting. The machine learning task is the first machine learning task, and the method may further include: modifying the operation of the first machine learning task on at least one of the one or more computing devices based on a priority of the first machine learning task and a priority of the second machine learning task, and allocating the second machine learning task to at least one of the one or more computing devices based on the step of modifying. The method may further include: determining an operating state of at least one of the one or more computing devices, and allocating the machine learning task to the data structure based on the operating state.At least one of the one or more computing devices may include a first computing device of the one or more computing devices, the characteristics of the machine learning task may include a first compatibility with the first computing device of the one or more computing devices, and a second compatibility with a second computing device of the one or more computing devices, and the method may further include: determining an operating state of the first computing device of the one or more computing devices, and allocating the machine learning task to the second computing device of the one or more computing devices based on the operating state and the second compatibility. The characteristics of the machine learning task may include a size of the machine learning task, and the step of allocating may include: allocating the machine learning task to at least one of the one or more computing devices based on the size of the machine learning task. At least one of the one or more computing devices may include a first computing device of the one or more computing devices, and the method may further include: modifying a characteristic of the computing system, and based on the step of modifying, allocating the machine learning task to the second computing device of the one or more computing devices. The characteristics of the computing system may include a policy. The characteristics of the computing system may include an operating parameter.

[0006] A system may include: at least one memory configured to store information for a machine learning task; a computing system including one or more computing devices; and at least one processor configured to: determine a characteristic of the machine learning task based on the information; determine a characteristic of the computing system; and assign the machine learning task to at least one of the one or more computing devices based on the characteristic of the machine learning task and the characteristic of the computing system. The characteristic of the computing system may include a policy, the characteristic of the machine learning task may include: a first compatibility with a first computing device in the one or more computing devices based on the policy, and a second compatibility with a second computing device in the one or more computing devices based on the policy, and the at least one processor may be configured to: assign the machine learning task to the first computing device in the one or more computing devices based on the policy and the first compatibility. The characteristic of the computing system may include a first policy and a second policy, the characteristic of the machine learning task may include a first compatibility with a first computing device in the one or more computing devices based on the first policy, and a second compatibility with a second computing device in the one or more computing devices based on the second policy, and the at least one processor may be configured to: assign the machine learning task to the first computing device in the one or more computing devices based on the first policy and the first compatibility.

[0007] An apparatus may include: at least one memory configured to store information for a computing task; a computing system including one or more computing devices; and at least one processor configured to: determine a characteristic of the computing task based on the information; determine a characteristic of the computing system; and assign the computing task to at least one of the one or more computing devices based on the characteristic of the computing task and the characteristic of the computing system. The characteristic of the computing task may include performance compatibility with at least one of the one or more computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings are not necessarily drawn to scale, and throughout the drawings, for illustrative purposes, elements of similar structure or function may generally be represented by the same reference numerals or portions thereof. The drawings are intended only to facilitate the description of the various embodiments described herein. The drawings do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. In order to prevent the drawings from becoming obscure, not all components, connections, etc. may be shown, and not all components may have reference numerals. However, the pattern of component configurations may be readily apparent from the drawings. The drawings, together with the specification, illustrate example embodiments of the present disclosure, and the drawings, together with the specification, are used to explain the principles of the present disclosure.

[0009] Figure 1 An embodiment of a computing system according to a disclosed example embodiment is shown.

[0010] Figure 2 An example embodiment of a computational workload implemented on a computer system according to the disclosed example embodiments is shown.

[0011] Figure 3 An embodiment of a scheme for distributing one or more computing tasks to one or more computing devices according to the disclosed example embodiments is shown.

[0012] Figure 4 An example embodiment of a scheme for distributing one or more computing tasks to one or more computing devices according to the disclosed example embodiments is shown.

[0013] Figure 5 A flow chart illustrating an embodiment of a method for allocating computing tasks to computing devices according to a disclosed example embodiment.

[0014] Figure 6 An example embodiment of a scheme for distributing one or more machine learning tasks to one or more computing devices according to the disclosed example embodiments is shown.

[0015] Figure 7 An embodiment of a computing system according to a disclosed example embodiment is shown.

[0016] Figure 8An example embodiment of a computing system according to the disclosed example embodiments is shown. DETAILED DESCRIPTION

[0017] A heterogeneous computing system may include various types of computing resources (such as one or more central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), etc.). Some artificial intelligence (AI) and / or machine learning (ML) applications (such as games, self-driving vehicles, augmented reality, etc.) may use multiple models that can perform various computing tasks (such as object detection, image recognition, image generation, etc.). For convenience, artificial intelligence and / or machine learning may be collectively and / or individually referred to as artificial intelligence and / or machine learning. In order to run a machine learning application on a heterogeneous computing system, one or more tasks may be assigned to one or more computing resources. However, assigning a task to a computing resource that is not suitable for the task may result in relatively low performance (e.g., throughput, latency, bandwidth, efficiency, etc.), high power consumption, high operating costs, etc.

[0018] The task allocation scheme according to the disclosed example embodiments may allocate machine learning tasks to computing devices based on one or more characteristics of the task, device, system in which the device can operate, etc. For example, a scheduler for a heterogeneous computing system with different types of computing resources may receive a request for performing an inference task using a machine learning model. The scheduler may use task allocation logic to allocate the inference task to a computing device that may be suitable for the task based on, for example, one or more characteristics of the inference task and / or one or more characteristics of the system.

[0019] Examples of task characteristics may include task compatibility (e.g., compatibility of the task with a particular type of computing device), task priority (e.g., a relative level of importance of the task compared to, for example, one or more other tasks, applications, etc.), and / or task size (e.g., the size of a model used by the task, the size of a data set used by the task, etc.).

[0020] Examples of system characteristics may include one or more system policies (e.g., regulating the system for performance, energy efficiency, etc.), one or more computing device topologies (e.g., characteristics of the devices in the system (such as, type, number, and / or capacity)), hardware utilization (e.g., busyness of one or more computing devices in the system (such as, percentage of capacity, such as throughput, bandwidth, etc.)), and / or system states or device states (e.g., power gating, thermal throttling, operating frequency, etc.).

[0021] In some embodiments, the task allocation scheme may use one or more data structures to determine one or more characteristics on which the task allocation is based. For example, the task allocation scheme may construct, maintain, use, etc. one or more data structures for information related to characteristics of tasks, applications, systems, devices, etc.

[0022] The information stored in the one or more data structures may be received, obtained, captured, etc. from various sources and / or using various techniques. For example, information about one or more task characteristics may be obtained from a task software context (e.g., an application context), received from a user (e.g., passed as a parameter of a task request), etc. Task characteristics may be received, obtained, etc. at any time (e.g., during system startup, when an application is loaded, when a task request is received, etc.).

[0023] As another example, information about one or more system characteristics may be obtained from a computing device, firmware (e.g., system firmware, device firmware, etc.), a system management controller, etc. The system characteristics may be obtained at system boot (e.g., by polling an agent (e.g., computing device) on a system bus, interconnect, etc.), to identify computing devices that may be installed in the system when hot-swapping computing devices, when loading and / or updating drivers for devices, etc.

[0024] In some embodiments, one or more data structures may be implemented in any form (e.g., table, tree, linked list, etc.). One or more data structures may be stored in any form and / or medium (such as software data structures stored in shared memory, hardware data structures stored in registers, etc.).

[0025] The present disclosure covers many aspects related to assigning computing tasks to computing devices. The aspects disclosed herein may have independent utility and may be embodied individually, and not every embodiment may utilize every aspect. In addition, aspects may also be implemented in various combinations, some of which may amplify some benefits of individual aspects in a collaborative manner.

[0026] For purposes of illustration, some embodiments may be described in the context of some specific implementation details, such as specific types of computing tasks, computing devices, computing resources, data connections, component configurations, etc. However, the disclosed aspects are not limited to the specific implementation details or any other implementation details.

[0027] In some figures, elements similar to elements shown in other figures may be indicated by reference numerals ending with and / or containing the same numbers, letters, etc. Multiple instances of elements identified using the same base number and different suffixes in the present disclosure may be indicated by the base number individually and / or collectively. For example, Figure 1The one or more computing devices 102a, 102b, ... shown in FIG. 1 may be individually and / or collectively referred to as (one or more) computing devices 102. As another example, Figure 2 The one or more inference tasks 221 - 1 , 221 - 2 , . . . shown in may be individually and / or collectively referred to as (one or more) inference tasks 221 .

[0028] Figure 1 An embodiment of a computing system according to a disclosed example embodiment is shown. Figure 1 The computing system 100 shown in FIG. 1 may include a system management controller 139 and one or more computing devices (such as one or more CPUs 102 a , GPUs 102 b , TPUs 102 g , NPUs 102 h , digital signal processors (DSPs) 102 j , FPGAs 102 k , etc.). Figure 1 Any or all of the components shown in may communicate using one or more communication connections 103 .

[0029] although Figure 1 The computing system 100 shown in the figure is not limited to any particular physical configuration, but in some embodiments, some or all of the components of the computing system 100 may be implemented with a system on chip (SoC). Additionally or alternatively, some or all of the components of the computing system 100 may be implemented with a system-in-package (SIP). For example, in some embodiments, one or more CPUs 102a, GPUs 102b, TPUs 102g, NPUs 102h, digital signal processors (DSPs) 102j, and / or FPGAs 102k may be implemented as integrated circuit (IC) chips (which may also be referred to as chiplets, dies, etc.), which are attached to a packaging substrate and are configured to communicate using one or more communication connections 103 that may be implemented with one or more die-to-die (D2D) interconnects. In such embodiments, the system management controller 139 may be implemented with a separate chip and / or as part of one or more IC chips (e.g., as part of one or more CPUs 102a). One or more CPUs 102a may be implemented as a CPU cluster (e.g., including one or more CPU cores 126). Additionally or alternatively, some or all components of the computing system 100 may be implemented in any other form including one or more personal computers, workstations, smart phones, tablet computers, servers, server chassis, modules, server racks, data rooms, data centers, edge data centers, mobile edge data centers, and / or any combination thereof, and in any form factor (e.g., adapter card form factor (e.g., PCIe (Peripheral Component Interconnect Express) adapter card), storage device form factor, etc.).

[0030] Figure 2 An example embodiment of a computational workload implemented on a computer system according to an example embodiment disclosed is shown. For illustrative purposes, the computational workload 210 is shown as a gaming workload having one or more computational tasks (which may also be referred to as tasks) implemented as one or more AI inference tasks. However, in other embodiments, the computational workload 210 may include any type of computational task for AI, ML, neural networks, deep learning, etc., such computational tasks including any type of one or more of the following items: algorithms, data movement, data management, data selection, filtering, encryption and / or decryption, compression and / or decompression, checksum calculations, hash value calculations, cyclic redundancy checks (CRCs), weight calculations, activation function calculations, training, inference (e.g., generative and / or predictive), classification, regression, clustering, etc.

[0031] Reference Figure 2 , the game computing workload 210 may include one or more computing tasks that may be implemented as AI inference tasks 221-1, 221-2, ... For example, a generative diffusion (e.g., artwork) task 221-1 (or generative diffusion 221-1) may generate visuals (such as background scenery), a system management task 221-2 (or system management 221-2) may control the overall operation of a game, a game system, etc., a non-player character (NPC) task 221-3 (or non-player character 221-3) may control the operation of one or more non-player characters that may interact with one or more players, and a generative environment task 221-4 (or generative environment 221-4) may generate non-player characters, situations, challenges, backgrounds, etc. An augmented reality (AR) task 221-5 (or augmented reality 221-5) may integrate one or more elements of the player's physical environment into the game, and / or a resolution scaling task 221-6 (or resolution scaling 221-6) may implement one or more scaling operations (e.g., zooming in and / or zooming out) for increasing and / or decreasing the resolution of an image to fit a screen space that may have a larger or smaller resolution than the image.

[0032] Other examples of AI workloads may include automobiles (e.g., autonomous vehicles), mobile devices, AR devices, edge computing applications, etc. In an embodiment of a self-driving vehicle workload, examples of AI inference tasks may include tasks for identifying the presence and / or state of traffic lights, tasks for detecting lines, lanes, etc. on a road, tasks for detecting other vehicles, tasks for detecting pedestrians, and / or tasks for making driving decisions based on one or more data sources (such as one or more types of sensor data) and / or outputs from other tasks.

[0033] AI inference tasks 221-1, 221-2, ... can be implemented using one or more inference models 227-1, 227-2, ... respectively. Examples of inference models 227 may include neural networks (e.g., convolution, feedforward, recursive, etc.), decision trees, random forests, regression (e.g., linear, logical, etc.), support vector machines, large language models, etc. Inference models 227-1, 227-2, ... can be run on one or more computing devices 202 (e.g., 202-1, 202-2, 202-3, 202-4, 202-5, 202-6, ...) in computing system 200.

[0034] However, determining which inference task 221 and / or inference model 227 to run on which computing device 202 can be challenging, particularly for workloads where multiple tasks and / or models can run simultaneously (such as games, autonomous vehicles, etc.). A task can have one or more device compatibilities (which can also be referred to as device affinities). In addition, a task can have one or more different types of compatibilities with one or more different types of computing devices. For example, a visual generation task can have performance (e.g., speed) compatibility with a GPU because the task can run relatively quickly (e.g., with the highest throughput) on a GPU. However, because the task can have relatively low power consumption (e.g., can run most efficiently) on an FPGA, the same task can have efficiency compatibility with an FPGA. As another example, a system management task that can perform a database lookup can have latency compatibility with a DPU because the DPU can retrieve requested data with relatively low latency (e.g., shortest latency).

[0035] Determining which inference task 221 and / or inference model 227 to run on which computing device 202 may also involve further challenges related to considering one or more characteristics of the computing system 200 and / or the computing device(s) 202. Examples of such considerations may include: the topology of the computing system 200 (e.g., the number, type, arrangement, etc.) of the computing device(s) 202, hardware utilization (e.g., the busyness of the computing device(s) 202), etc. In addition, the computing workload 210 may be created, programmed, etc. without knowledge of the topology of the computing system 200 on which it may be executed. For example, a gaming workload may be designed for a specific (e.g., fixed) hardware configuration (such as a gaming computer with a CPU and a GPU), but may ultimately be executed on a different computing device (such as an available SoC and / or SIP (such as a gaming computer with a CPU and a GPU)). Figure 1 The SoC and / or SIP shown and / or described is executed on a computing system of a smart phone.

[0036] Figure 3An embodiment of a scheme for distributing one or more computing tasks to one or more computing devices according to an example embodiment disclosed is shown. Figure 3 In the scheme 330 shown in the figure, the allocation logic 304 can allocate one or more computing tasks 321 (e.g., 321-1, 321-2, ...) in the computing workload 310 to one or more computing devices 302 (e.g., 302-1, 302-2, ...) in the computing system 300 based on one or more task characteristics 328 and / or one or more system characteristics 329.

[0037] Examples of task characteristics 328 may include one or more device compatibilities (e.g., performance compatibility, efficiency compatibility, latency compatibility), task size (e.g., size of a model used to perform the task, amount of data used as input to the task and / or output from the task, etc.), task type (e.g., training, inference, data movement, data selection, compression, encryption, etc.), task priority, etc. In some embodiments, device compatibility may be represented as one or more lists (possibly ranked and / or possibly based on type of compatibility) of one or more devices on which the task may be performed.

[0038] Examples of system characteristics 329 may include one or more system policies (e.g., performance policy, efficiency policy, balancing policy, etc.), system topology (e.g., the number, type, arrangement, etc. of (one or more) computing devices 302), one or more operating states (e.g., a busy state, a power gating state, a thermal throttling state, etc. of one or more computing devices and / or communication connections between devices), operating parameters (e.g., a clock speed of one or more devices), hardware utilization (e.g., a percentage of computing resources, capacity, etc. of one or more computing devices 302), and the like.

[0039] Although allocation logic 304 is not limited to any particular implementation details, in some embodiments, allocation logic 304 can be implemented with a scheduler (such as an operating system scheduler). In some embodiments, some or all of the task allocation functions associated with allocation logic 304 can be implemented with an extension, modification, plug-in, etc. of an operating system scheduler.

[0040] In some embodiments, Figure 3 The task allocation scheme 330 shown in the figure may employ one or more computing task compatibilities, which may improve the operation of the computing system 300 and / or one or more computing devices 302 depending on the implementation details (e.g., by matching one or more computing tasks 321 with one or more computing devices 302 on which the computing tasks 321 may have improved (e.g., optimized) performance, efficiency, latency, etc.).

[0041] Additionally or alternatively, the task allocation scheme 330 may improve utilization of the computing system 300 and / or one or more computing devices 302 , for example, by providing a flexible scheme for allocating one or more computing tasks 321 to a heterogeneous system topology.

[0042] Additionally or alternatively, the task allocation scheme 330 can improve the operation of the computing system 300 and / or one or more computing devices 302 by enabling the operation of the system 300 and / or one or more computing devices 302 to be adjusted to a specific operating goal (such as a performance goal, an efficiency (e.g., power consumption) goal, etc.). For example, in some embodiments, a system policy can be modified (e.g., during the execution of one or more computing tasks 321) by changing from a first (e.g., performance) policy to a second (e.g., efficiency) policy. Based on the change in system policy, one or more computing tasks 321 can be reassigned to different one or more computing devices 302 to implement the modified system policy (e.g., to adjust the computing system 300 and / or one or more computing devices 302 to the modified operating goal).

[0043] Figure 4 An example embodiment of a scheme for distributing one or more computing tasks to one or more computing devices according to the disclosed example embodiments is shown. Figure 4 The scheme 430 shown in FIG. 4 can be used to implement any allocation scheme disclosed herein, including the allocation schemes described with respect to other figures, and / or can be implemented with any allocation scheme disclosed herein, including the allocation schemes described with respect to other figures, in which similar elements can be indicated by reference numerals ending with and / or containing the same numbers, letters, etc. For example, Figure 4 The scheme 430 shown in FIG. 4 can be used to implement Figure 3 The computing task allocation scheme 330 is shown in FIG.

[0044] Reference Figure 4 The task allocation scheme 430 may include allocation logic 404, which may allocate one or more computing tasks 421 in the computing workload 410 to one or more computing devices 402 in the computing system 400 based on one or more task characteristics 428 and / or one or more system characteristics 429.

[0045] Examples of task characteristics 428 may include device compatibility, task priority, task size, task type, etc. For example, the device compatibility of a particular computing task may include a device compatibility data structure having a first list of devices with which the task may have performance compatibility. The first list may include a first entry for a first device (e.g., typically, a GPU or a first type, a first model, etc. of a GPU) that may indicate that the task may operate on the first device at a first performance (e.g., best performance or fastest performance), a second entry for a second device (e.g., typically, a second type, a second model, etc. of an NPU or GPU) that may indicate that the task may operate on a second device at a second performance (e.g., second best performance or second fastest performance), and a third entry for a third device (e.g., typically, a third type, a third model, etc. of an FPGA or GPU) that may indicate that the task may operate on the third device at a third performance (e.g., third best performance or third fastest performance).

[0046] In some embodiments, the device compatibility data structure may include a second list of devices with which the task may have efficiency compatibility. The second list may include: a first entry for a first device (e.g., typically, a first type, a first model, a first configuration (e.g., a first program), etc. of an FPGA or FPGA) that may indicate that the task may operate at a first efficiency (e.g., the best efficiency or the highest efficiency) on the first device, a second entry for a second device (e.g., typically, a second type, a second model, a second configuration (e.g., a second program), etc. of a CPU or FPGA) that may indicate that the task may operate at a second efficiency (e.g., the second best efficiency or the second highest efficiency) on the second device, and a third entry for a third device (e.g., typically, a third type, a third model, a third configuration (e.g., a third program), etc. of a GPU or FPGA) that may indicate that the task may operate at a third efficiency (e.g., the third best efficiency or the third highest efficiency) on the third device.

[0047] Similarly, in some embodiments, the device compatibility data structure may include a third list of devices with which the task may have latency compatibility (eg, different latency).

[0048] Examples of task priorities may include a numerical value (e.g., an integer) indicating the importance (e.g., relative importance) of the task. For example, in a gaming application (e.g., a gaming workload), a generative environment task may have a relatively high priority to prevent stuttering in the gaming environment. As another example, a first task that has one or more data dependencies with the first task (e.g., one or more other tasks may use output data from the first task as input data) may have a relatively high priority to prevent one or more other tasks from waiting while the first task is executing.

[0049] Examples of task types may include any type of algorithm for AI, ML, neural networks, deep learning, etc. (e.g., engineering computing and / or scientific computing), modeling (e.g., 2D and / or 3D modeling for artwork, design, etc.), audio and / or video processing (e.g., speech and / or image recognition, resolution scaling, etc.), augmented reality, data movement, data management (e.g., database management), data selection, filtering, encryption and / or decryption, compression and / or decompression, checksum calculation, hash value calculation, cyclic redundancy check (CRC), weight calculation, activation function calculation, training, inference (e.g., generative and / or predictive), classification, regression, clustering, etc.

[0050] The task size can, for example, be based at least in part on the size of the model (e.g., the size of the neural network, the number of neurons, the number of layers, the number of connections, the number of activation functions, the number of weights, the number of nodes, the number of edges, the complexity of the algorithm (e.g., regression equation), the size of the decision tree, the size of the random forest, etc.). As another example, the task size can be based at least in part on the amount of input data used by the task, the amount of intermediate data used by the task (e.g., in scratchpad memory), the amount of output data generated by the task, etc.

[0051] Examples of system characteristics 429 may include one or more system policies, system (eg, device) topology, one or more operating states, parameters, utilization, etc., of computing system 400 and / or one or more computing devices 402 , etc.

[0052] Examples of system policies may include a performance policy in which the allocation logic 404 may attempt to improve (e.g., maximize or optimize) the overall performance (e.g., throughput) of one or more (e.g., all) computing devices 402 in the computing system 400. Another example may include an efficiency (e.g., power conservation) policy in which the allocation logic 404 may attempt to improve the energy efficiency (e.g., reduce power consumption) of one or more computing devices 402 in the computing system 400. Further examples may include a balancing policy in which the allocation logic 404 may attempt to maintain the overall performance (e.g., throughput) of the computing system 400 and / or one or more computing devices 402 at a particular level (e.g., a target performance level) while preventing the power consumption of the computing system 400 and / or one or more computing devices 402 from exceeding a particular level (e.g., a maximum power consumption level).

[0053] In some embodiments, the system topology may include one or more numbers, types, arrangements, etc. of computing devices 402 in computing system 400. Example system topologies may include Figure 1 The number and / or arrangement of computing devices 102 shown in Figure 1The topology (eg, number, type, arrangement, etc.) of the communication connections 103 shown in FIG.

[0054] Examples of operating states may include utilization (e.g., busyness) of computing system 400 and / or one or more computing devices 402, where utilization may be represented as, for example, one or more binary values ​​(e.g., busy or not busy), one or more percentages (e.g., a percentage of overall computing resources of computing system 400 that are available, a percentage of computing resources (e.g., cores) of computing devices 402 that are available, a percentage of available data transfer bandwidth to and / or from system 400 and / or computing devices 402, etc.).

[0055] Additional examples of operating states may include power gating states, thermal throttling states, etc. of computing system 400 and / or one or more computing devices 402 and / or communication connections to, from, and / or between computing devices 402 .

[0056] Examples of operating parameters may include clock speeds of one or more computing devices 402. For example, in some embodiments, allocation logic 404 and / or a system management controller (e.g., Figure 1 The system management controller 139 shown in FIG. 1 may reallocate one or more computing tasks 421 and / or adjust one or more clock speeds of one or more computing devices 402 to adjust the computing system 400 to achieve a target performance level and / or a target energy efficiency.

[0057] Any of the task characteristics 428 and / or system characteristics 429 may be stored in, managed by, etc., software, hardware, or a combination thereof. For example, in some embodiments, one or more task characteristics 428 may be stored in one or more data structures in a task software context that may be allocated by an operating system, system-level software, etc. In such embodiments, the operating system, system-level software, etc. may collect and / or populate the one or more task characteristics 428 manually, programmatically (e.g., using an application programming interface (API)), etc. using a user interface (e.g., a configuration utility). In some embodiments, the system management controller may collect (e.g., capture) the one or more task characteristics 428 by searching for code in one or more applications, computing workloads, computing tasks, etc., and / or observing one or more operations of one or more applications, computing workloads, computing tasks, etc.

[0058] As another example, one or more system characteristics 429 may be stored in firmware (e.g., system firmware, device firmware, etc.), in one or more hardware structures (such as one or more registers) (e.g., in a SoC, SIP, etc.), in a shared memory area that can be managed and / or accessed (e.g., read and / or written) by hardware, software, firmware, or a combination thereof. In some embodiments, one or more system characteristics 429 may be collected and / or populated manually, programmatically (e.g., using an application programming interface (API)), etc. using a user interface (e.g., a configuration utility). In some embodiments, a system management controller may collect (e.g., capture) one or more system characteristics 429 by searching the code of one or more computing devices (e.g., one or more device drivers) and / or observing one or more operations of one or more computing devices.

[0059] exist Figure 4 In the embodiment shown in , the allocation logic 404 may include a task queue 431, a ready queue 432, a scheduling algorithm 433, a selection logic 435, and / or a wait queue 434, which may be collectively referred to and / or characterized as a scheduler and / or scheduling logic. Any or all of the task queue 431, the ready queue 432, and / or the wait queue 434 may be implemented with any type of configuration, such as first-in, first-out (FIFO), first-in, last-out (FILO) (e.g., a stack), a circular queue, etc.

[0060] In some embodiments, task queue 431 may be used to store one or more of computing tasks 421 that may not be ready for execution. For example, computing task 421 may have operational dependencies and / or data dependencies on another computing task 421, user input, input data received from outside computing system 400, etc. Allocation logic 404 may store computing task 421 in task queue 431 until the dependency is resolved (e.g., the data on which computing task 421 depends becomes available), and move computing task 421 from task queue 431 to ready queue 432 based on resolving the dependency.

[0061] In some embodiments, the ready queue 432 may hold one or more computing tasks 421 that are ready to be executed on one or more computing devices 402 .

[0062] The scheduling algorithm 433 may determine (or select) an order for scheduling one or more computing tasks 421 stored in the ready queue 432. The scheduling algorithm 433 may be implemented, for example, using a first-come, first-served (FCFS) algorithm, in which the one or more computing tasks 421 may be scheduled in the order in which they are placed in the ready queue 432 (e.g., the ready queue 432 is implemented as a FIFO queue). As another example, the scheduling algorithm 433 may be implemented with a priority scheduling algorithm, in which the one or more computing tasks 421 may be scheduled in an order based on their relative priorities (e.g., the task with the highest priority may be scheduled next). As another example, the scheduling algorithm 433 may be implemented with a round robin scheduling algorithm, in which one or more computing tasks 421 may be scheduled by rotating between entries in the ready queue 432 (e.g., without regard to order of placement, priority, etc.), or by rotating between sources of entries in the ready queue 432 (e.g., by rotating between one or more applications, processes, virtual machines, etc. that may place entries in the ready queue 432). As yet another example, the scheduling algorithm 433 may be implemented with a shortest task first (STN) (which may also be referred to as a shortest job first (SJN)) algorithm, in which tasks with the smallest size may be scheduled next.

[0063] For purposes of illustration, it may be assumed that scheduling algorithm 433 determines that compute task 421 - 1 is the next compute task to be assigned (eg, scheduled).

[0064] Based on the allocation order determined by the scheduling algorithm 433 (e.g., once the next computing task 421-1 to be scheduled has been determined), the selection logic 435 (which may also be referred to as placement logic, arbitration logic, and / or distribution logic) may select the computing device 402 to which the next task 421-1 is to be allocated. The selection logic 435 may select the computing device 402 to which the next task 421-1 is to be allocated based on one or more task characteristics 428 of the next task 421-1, one or more system characteristics 429 of the computing system 400, and / or a combination of one or more task characteristics 428 of the next task 421-1 and one or more system characteristics 429 of the computing system 400, etc. For example, in the case where the computing system 400 may implement a performance policy, the selection logic 435 may determine the performance compatibility of the next task 421-1 by accessing a list of computing devices 402 having performance compatibility with the next task 421-1. If the list has more than one computing device 402 that has performance compatibility with the next task 421-1, the selection logic 435 may select the computing device 402 that has relatively high (e.g., best) performance compatibility with the next task 421-1. In some embodiments, the selection logic 435 may ignore the efficiency compatibility and / or delay compatibility of the next task 421-1 in the case where the computing system 400 may implement a performance policy. Similarly, the selection logic 435 may ignore the performance compatibility and / or delay compatibility of the next task 421-1 in the case where the computing system 400 may implement an efficiency policy.

[0065] For purposes of illustration, it may be assumed that the scheduling algorithm 433 selects the computing device 402 - 1 to which to assign (eg, schedule the next task 421 - 1 ).

[0066] As indicated by arrow 436, based on selecting the computing device 402-1, the allocation logic 404 may allocate the next task 421-1 to the selected computing device 402-1 (e.g., may schedule the next task 421-1 onto the selected computing device 402-1). The allocation logic 404 may allocate and / or schedule the next task 421-1, for example, by sending one or more instructions, indications, etc. to the computing device 402-1, by compiling code for the computing task 421-1 on the selected computing device 402-1, by notifying an execution environment of the allocation, by notifying a scheduler (e.g., an operating system) of the allocation, by placing the computing task 421-1 in a queue (e.g., of the computing device 402-1), and / or in any other suitable manner.

[0067] Additionally or alternatively, the allocation logic 404 may use one or more additional task characteristics 428 of the next task 421-1 and / or system characteristics 429 of the system 400 to determine how to allocate the next task 421-1. For example, as shown by arrow 437, based on the selected computing device 402-1 being busy and / or currently executing a computing task with a higher priority than the next task 421-1, the allocation logic 404 may allocate the next task 421-1 to a data structure for storing machine learning tasks (e.g., storing the next task 421-1 in the waiting queue 434). As shown by arrow 438, when the current task and / or the higher priority task is completed by the selected computing device 402-1, the allocation logic 404 may schedule the next task 421-1 to the selected computing device 402-1. Additionally or alternatively, as shown by arrow 440, for example, after a timeout period spent in the waiting queue, the allocation logic 404 may move the next task 421-1 from the waiting queue 434 to the ready queue 432.

[0068] Additionally or alternatively, based on the priority of the computing task currently being executed by the selected computing device 402-1 and the priority of the next task 421-1, the allocation logic 404 may modify the operation of the currently executed computing task on the selected computing device 402-1. For example, based on the selected computing device 402-1 currently executing a computing task having a lower priority than the next task 421-1, the allocation logic 404 may preempt the lower priority task, for example, by moving the lower priority task to the waiting queue 434 as shown by arrow 441 and scheduling the next task 421-1 to the selected computing device 402-1 as shown by arrow 436. As shown by arrow 438, when the (higher priority) next task 421-1 is completed by the selected computing device 402-1, the allocation logic 404 may move the preempted computing task from the waiting queue 434 back to the selected computing device 402-1.

[0069] Additionally or alternatively, based on the selected computing device 402-1 being busy and / or currently executing a computing task having a higher priority than the next task 421-1, the allocation logic 404 may determine devices of different performance compatibilities for the next task 421-1. For example, the allocation logic 404 may select a computing device 402-2 having a relatively low performance compatibility (e.g., lower performance) with the next task 421-1, and schedule the next task 421-1 to the selected computing device 402-2.

[0070] Table 1 shows pseudo code for an example embodiment of a computing task allocation scheme according to an example embodiment disclosed. The pseudo code shown in Table 1 can be used to implement any allocation scheme disclosed herein, including the allocation scheme described with respect to any figure, and / or can be implemented with any allocation scheme disclosed herein, including the allocation scheme described with respect to any figure. For example, the pseudo code shown in Table 1 can be used to implement Figure 3 The computing task allocation scheme 330 shown in FIG. Figure 4 Although the embodiments of the computing task allocation scheme described with respect to the pseudo code shown in Table 1 are not limited to any specific implementation details, one or more aspects of the pseudo code shown in Table 1 may refer to the embodiments of the computing task allocation scheme described with respect to Figure 5 The flowcharts of embodiments of the computing task allocation scheme shown and / or described are understood.

[0071] Table 1 Figure 5 A flow chart of an embodiment of a method for allocating computing tasks to computing devices according to a disclosed example embodiment is shown. Figure 5 The method 545 shown in FIG. 545 may be used to implement any allocation scheme disclosed herein, including the allocation schemes described with respect to other figures and / or Table 1, or may be implemented with any allocation scheme disclosed herein, including the allocation schemes described with respect to other figures and / or Table 1. For illustrative purposes, Figure 5 The method shown in Figure 4 430 and / or the pseudo code shown in Table 1, but Figure 5 The methods presented are not limited to these or any other implementation details.

[0072] Reference Figure 5 , the method may begin at operation 545-1, where the method may determine whether the computing system is implementing a first-come, first-served (FCFS) scheduling algorithm (e.g., row 1 of Table 1). If the scheduling algorithm is FCFS, the method may proceed to operation 545-2, where the method may initialize a task_to_be_scheduled variable with the first task (e.g., first_task) placed in the ready queue 432 (e.g., row 2 of Table 1), and proceed to operation 545-8. If the scheduling algorithm is not FCFS, the method may proceed to operation 545-3.

[0073] At operation 545-3, the method may determine whether the computing system is implementing a shortest task first (STN) scheduling algorithm (e.g., row 3 of Table 1). If the scheduling algorithm is STN, the method may proceed to operation 545-4, where the method may initialize the task_to_be_scheduled variable with the smallest size task (e.g., smallest_task) in the ready queue 432 (e.g., row 4 of Table 1), and proceed to operation 545-8. If the scheduling algorithm is not STN, the method may proceed to operation 545-5.

[0074] At operation 545-5, the method may determine whether the computing system is implementing a priority scheduling algorithm (e.g., row 5 of Table 1). If the scheduling algorithm is priority-based, the method may proceed to operation 545-6, where the method may initialize the task_to_be_scheduled variable with the highest priority task (e.g., highest_priority_task) in the ready queue 432 (e.g., row 6 of Table 1), and proceed to operation 545-8. If the scheduling algorithm is not priority-based, the method may proceed to operation 545-7.

[0075] At operation 545 - 7 , the method may initialize the task_to_be_scheduled variable (eg, row 8 of Table 1) with the next task (eg, next_task) in the ready queue 432 , for example, using a round-robin scheduling algorithm that may be used as a default scheduling algorithm.

[0076] At operation 545 - 8 , the method may initialize the task_priority variable (eg, row 9 of Table 1) with the priority of task_to_be_scheduled (eg, priority_of_task_to_be_scheduled).

[0077] At operation 545-9, the method may determine whether the computing system 400 is implementing a performance policy (e.g., row 10 of Table 1). If the computing system 400 is implementing a performance policy, the method may proceed to operation 545-10 (e.g., row 11 of Table 1), where the method may determine whether the computing system 400 includes a computing device 402 that has performance compatibility with task_to_be_scheduled (e.g., performance_compatible_device_exists=true). The method may determine whether the computing system 400 includes a computing device 402 that has performance compatibility with task_to_be_scheduled (e.g., performance_compatible_device_exists=true), for example, by checking task_to_be_scheduled (e.g., Figure 4The determination is performed based on a performance compatibility list of the next task 421-1 in task_to_be_scheduled and / or a system topology data structure of computing system 400. In some embodiments, if the performance compatibility list of task_to_be_scheduled includes more than one computing device, and the first computing device in the list does not exist in computing system 400, the method may check whether a different computing device in the list exists in computing system 400.

[0078] If the method determines at operation 545-10 that the computing system 400 includes a computing device having performance compatibility with task_to_be_scheduled (eg, Figure 4 If the computing device 402-1 in the example is selected, the method may proceed to operation 545-11, at which the method may initialize the selected_device variable with a performance-compatible device (eg, performance_compatible_device) (eg, row 12 of Table 1), and proceed to operation 545-16.

[0079] However, if at operation 545-10 the method determines that the computing system 400 does not include a computing device having performance compatibility with task_to_be_scheduled, the method may proceed to operation 545-15 where the method may initialize the selected_device variable with a default computing device (eg, default_device) (eg, row 14 of Table 1).

[0080] If the method determines at operation 545-9 that the computing system 400 is not implementing a performance policy, the method may proceed to operation 545-12, where the method may determine whether the computing system 400 is implementing an efficiency policy (e.g., row 15 of Table 1). If the computing system 400 is implementing an efficiency policy, the method may proceed to operation 545-13, where the method may determine whether the computing system 400 includes a computing device 402 that is efficiency compatible with task_to_be_scheduled (e.g., row 16 of Table 1). The method may determine whether the computing system 400 includes a computing device 402 that is efficiency compatible with task_to_be_scheduled, for example, by checking a value for task_to_be_scheduled (e.g., row 17 of Table 1). Figure 4 The determination is performed based on an efficiency compatibility list for the next task 421-1 in task_to_be_scheduled and / or a system topology data structure for computing system 400. In some embodiments, if the efficiency compatibility list for task_to_be_scheduled includes more than one computing device, and the first computing device in the list does not exist in computing system 400, the method may check whether a different computing device in the list exists in computing system 400.

[0081] If the method determines at operation 545-13 that the computing system 400 includes computing devices having efficiency compatibility with task_to_be_scheduled (eg, Figure 4 ), the method may proceed to operation 545-14, where the method may initialize the selected_device variable with an efficiency compatible device (eg, efficiency_compatible_device) (eg, row 17 of Table 1), and proceed to operation 545-16.

[0082] However, if at operation 545 - 13 the method determines that the computing system 400 does not include a computing device having efficiency compatibility with task_to_be_scheduled, the method may proceed to operation 545 - 15 where the method may initialize the selected_device variable with a default computing device (eg, row 14 of Table 1).

[0083] At operation 545-16, the method may determine whether selected_device is busy (eg, row 20 of Table 1). If selected_device is not busy, the method may proceed to operation 545-17 (eg, row 27 of Table 1), where the method may schedule task_to_be_scheduled (eg, Figure 4 The next task 421-1 in the scheduler is assigned to the selected_device (eg, Figure 4 The computing device 402-1 in FIG.

[0084] However, if the method determines that selected_device is busy at operation 545-16, the method may proceed to operation 545-18, at which the method may compare the priority of the task currently executing on selected_device with the priority of task_to_be_scheduled (e.g., row 23 of Table 1). If the priority of the task currently executing on selected_device is lower than the priority of task_to_be_scheduled, the method may proceed to operation 545-19 (e.g., row 24 of Table 1), at which the method may preempt the currently executing task (e.g., by placing the currently executing task in the waiting queue 434). The method may proceed to operation 545-17 (e.g., row 27 of Table 1), at which the method may assign task_to_be_scheduled to the selected_device that may begin executing task_to_be_scheduled.

[0085] However, if the method determines at operation 545-18 that the priority of the task currently executing on selected_device is higher than or equal to the priority of task_to_be_scheduled, the method may proceed to operation 545-20 (e.g., row 26 of Table 1), at which the method may wait until the task currently executing on selected_device is completed. Upon completion of the currently executing task, the method may proceed to operation 545-17, at which the method may assign task_to_be_scheduled to selected_device, which may begin executing task_to_be_scheduled.

[0086] In some embodiments, the method may proceed from operation 545 - 17 to operation 545 - 1 and repeat method 545 (eg, continuously).

[0087] In some embodiments, the method may determine whether computing system 400 includes a second computing device having compatibility with task_to_be_scheduled instead of waiting at operation 545 - 20 , and if the method may determine that computing system 400 includes a second computing device having compatibility with task_to_be_scheduled , assign task_to_be_scheduled to the second computing device.

[0088] In some embodiments, the method 545 may include one or more operations for allocating tasks_to_be_scheduled to computing devices based on the size of the tasks_to_be_scheduled.

[0089] In some embodiments, method 545 may include one or more operations for modifying a policy, clock speed, etc. of computing system 400 and reallocating task_to_be_scheduled to different computing devices based on the modified policy, clock speed, etc.

[0090] Figure 6 An example embodiment of a scheme for distributing one or more machine learning tasks to one or more computing devices according to the disclosed example embodiments is shown. Figure 6 The scheme 630 shown in FIG. 630 may be used to implement any of the allocation schemes disclosed herein, including the allocation schemes described with respect to other figures, and / or may be implemented with any of the allocation schemes disclosed herein, including the allocation schemes described with respect to other figures, in which similar elements may be indicated by reference numerals ending with and / or including the same numbers, letters, etc. For example, Figure 6 The scheme 630 shown in FIG. 6 can be used to implement Figure 3 An example embodiment of a computing task allocation scheme 330 in which a computing workload 310 may be implemented as a gaming AI workload is shown in FIG.

[0091] Reference Figure 6 , similar to Figure 1 The game AI workload shown and / or described may include the game AI workload 610 and Figure 2 One or more AI inference tasks 621-1, 621-2, ... are similar to the inference tasks 221-1, 221-2, ... shown in FIG. One or more of the inference tasks 621-1, 621-2, ... may use one or more inference models 627-1, 627-2, ... respectively. The inference task allocation scheme 630 may include an allocation logic 604, which may be used, for example, to Figure 4 The schedulers shown and / or described above may be implemented with schedulers similar to those shown and / or described above. One or more computing devices may be configured as inference accelerators 602-1, 602-2, . . .

[0092] The allocation logic 604 may allocate one or more of the inference tasks 621-1, 621-2, ... to one or more inference accelerators 602-1, 602-2, ... that may respectively execute one or more inference models 627-1, 627-2, ... corresponding to the inference tasks 621-1, 621-2, .... The allocation logic 604 may allocate the inference tasks 621 based on one or more inference task characteristics 628 (which may be at least partially managed by software) and / or one or more system characteristics 629 (which may be at least partially managed by hardware).

[0093] Depending on implementation details, the inference task allocation scheme 630 may utilize one or more inference task device compatibilities to allocate one or more of the inference tasks 621 to one or more inference accelerators 602, which may have improved (e.g., optimized) power characteristics and / or performance characteristics for the specific inference tasks assigned to them. Additionally or alternatively, the inference task allocation scheme 630 may improve (e.g., maximize) hardware utilization, for example, by providing flexible placement of one or more inference tasks 621 on a computing system 600 with varying (e.g., heterogeneous) accelerator topologies. Additionally or alternatively, the inference task allocation scheme 630 may enable the operation of the computing system 600 and / or one or more inference accelerators 602 to be adjusted to one or more specific power targets and / or performance targets.

[0094] Figure 7 An embodiment of a computing system according to a disclosed example embodiment is shown. Figure 7 The computing system 700 shown in the figure can be used to implement any distribution scheme disclosed herein, including the distribution schemes described with respect to other figures, and / or be implemented with any distribution scheme disclosed herein, including the distribution schemes described with respect to other figures, in which similar elements can be indicated by reference numerals ending with the same numbers, letters, etc. and / or containing the same numbers, letters, etc.

[0095] Figure 7 The computing system 700 shown in FIG. 7 may include a host 701, one or more computing devices 702, and / or a system management controller 739 that may be configured to communicate using one or more communication connections 703. In some embodiments, the host 701 may be implemented with more than one host that may be individually and / or collectively referred to as the host 701.

[0096] The system management controller 739 may include a system that can implement the disclosed Figure 1 Any or all logic included in the system manager 139 shown and / or described and / or directed to Figure 3 , Figure 4 and / or Figure 6Any or all logic of any computing task allocation scheme of any of the allocation logic 304, 404, and / or 604 shown and / or described. Any or all logic implemented with the system management controller 739 or any or all logic used to implement the system management controller 739 may be located in a manner including at least partially in such as Figure 7 , at the host 701, at one or more computing devices 702, and / or at any other location or combination thereof (eg, distributed across different components).

[0097] against Figure 7 Any components shown and / or described may be arranged individually and / or in groups at any location (such as one or more nodes, servers, chassis, racks, clusters, data centers, edge data centers, etc.). In some embodiments, an arrangement of one or more devices (whether individually, located in a node, etc.) may be referred to as and / or may be characterized as a cluster.

[0098] Although the one or more communication connections 703 may be shown as being separate from other components, in some embodiments, one or more other components may be integrated with and / or configured within the one or more communication connections 703, between one or more other components using the one or more communication connections 703, etc. For example, in some embodiments, the system management controller 739 may be located between portions of the one or more communication connections 703.

[0099] The one or more communication connections 703 may use any type of interface, protocol, etc. to implement one or more interconnects, one or more networks, a network of networks (e.g., the Internet), etc., or a combination thereof, and / or be implemented with one or more interconnects, one or more networks, a network of networks (e.g., the Internet), etc., or a combination thereof. For example, the one or more communication connections 703 may implement any type of wired and / or wireless communication medium, interface, network, interconnect, protocol, etc., and / or be implemented with any type of wired and / or wireless communication medium, interface, network, interconnect, protocol, etc., including: PCIe, NVMe, NVMe over Fabric (NVMe-oF), Compute Express Link (CXL) and / or coherence protocols (such as, CXL.mem, CXL.cache, CXL.io, etc.), Gen-Z, Open Coherent Accelerator Processor Interface (Open CAPI), Cache Coherent Interconnect for Accelerators (CCIX), Advanced Extensible Interface (AXI), Direct Memory Access (DMA), Remote DMA (RDMA), RDMA over Converged Ethernet (ROCE). Advanced Message Queuing Protocol (AMQP), Ethernet, Transmission Control Protocol / Internet Protocol (TCP / IP), Fibre Channel, InfiniBand, Serial ATA (Advanced Technology Attachment) (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), iWARP, any generation of wireless network including 2G, 3G, 4G, 5G, 6G, etc., any generation of Wi-Fi, Bluetooth, Near Field Communication (NFC), etc., or any combination thereof. In some embodiments, the communication connection 703 may include one or more switches, hubs, nodes, routers, etc.

[0100] The host 701 may be implemented with any component or combination of components that may utilize one or more characteristics of the computing device 702. For example, the host may be implemented with one or more of a server, a storage node, a computing node, a central processing unit (CPU), a workstation, a personal computer, a tablet computer, a smart phone, etc., or a plurality thereof and / or a combination thereof. In some embodiments, the host 701 may include one or more communication interfaces 705 that may be used to implement any or all of the one or more communication connections 703.

[0101] In some embodiments, host 701 may be a source of one or more computing workloads 710 having one or more computing tasks that may be distributed to one or more computing devices 702 , for example, by system management controller 739 .

[0102] The computing device 702 may include a communication interface 705, a memory 707 (some or all of which may be referred to as device memory), one or more computing resources 706 (which may also be referred to as computing resources), a device controller 708, and / or device functional circuitry 709. The device controller 708 may control the overall operation of the computing device 702, including any operations, any features, etc. described herein. For example, in some embodiments, the device controller 708 may use one or more computing resources 706 to perform one or more computing tasks received from the host 701.

[0103] The communication interface 705 (which in some embodiments may be implemented with multiple communication interfaces 705 ) may be used to implement any or all of the one or more communication connections 703 .

[0104] The device function circuit 709 may include any hardware for implementing the main functions of the computing device 702. For example, if the computing device 702 is implemented as a storage device (e.g., a computing storage device), the device function circuit 709 may include a storage medium (such as a magnetic medium (e.g., in the case where the computing device 702 is implemented as an HDD or a tape drive), a solid-state medium (e.g., one or more flash memory devices), an optical medium, etc.). For example, in some embodiments, the storage device may be at least partially implemented as an SSD based on non-AND (NAND) flash memory, a persistent memory (PMEM) (such as a cross-grid non-volatile memory), a memory with a body resistance change, a phase change memory (PCM), or any combination thereof. In an embodiment where the computing device 702 is implemented as a storage device, the device controller 708 may include a media translation layer (such as a flash translation layer (FTL)) for interfacing with one or more flash memory devices. In some embodiments, the computing storage device may be implemented as a computing storage drive, a computing storage processor (CSP), and / or a computing storage array (CSA).

[0105] As another example, if the computing device 702 is implemented as a network interface controller (NIC) (e.g., a network interface card), the device function circuit 709 may include one or more modems, a network interface, a physical layer (PHY), a media access control layer (MAC), etc. As another example, if the computing device 702 is implemented as an accelerator, the device function circuit 709 may include one or more accelerator circuits, memory circuits, etc.

[0106] The one or more computing resources 706 may be implemented with any component or combination of components that can perform operations on data that can be received, stored, and / or generated at the computing device 702. Examples of computing resources may include combinatorial logic, sequential logic, timers, counters, registers, state machines, complex programmable logic devices (CPLDs), FPGAs, application specific integrated circuits (ASICs), embedded processors, microcontrollers, CPUs (such as complex instruction set computer (CISC) processors (e.g., x86 processors)) and / or reduced instruction set computer (RISC) processors (such as ARM processors) that can execute instructions stored in any type of memory and / or implement any type of execution environment (such as containers, virtual machines, operating systems (such as Linux), extended Berkeley packet filter (eBPF) environments), etc.), or combinations thereof.

[0107] The memory 707 may be used, for example, by one or more of the computing resources 706 to store input data, output data (e.g., calculation results), intermediate data, transitional data, etc. The memory 707 may be implemented, for example, with volatile memory (such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and any other type of memory (such as non-volatile memory).

[0108] In some embodiments, the memory 707 and / or computing resources 706 may include software, instructions, programs, code, etc. that can be executed, run, etc. using one or more computing resources (e.g., hardware (HW) resources). Examples may include software implemented in any language (such as, assembly language, C, C++, etc.), binary code, FPGA code, one or more operating systems, kernels, environments (such as, eBPF), etc. The software, instructions, programs, code, etc. may be stored in, for example, the memory 707 and / or computing resources 706. The software, instructions, programs, code, etc. may be downloaded, uploaded, side-loaded, pre-installed, built-in, etc. to the memory 707 and / or computing resources 706. In some embodiments, the computing device 702 may receive one or more instructions, commands, etc. for selecting, enabling, activating, executing, etc. the “software, instructions, programs, code, etc.” Examples of computing operations, functions, etc. that may be implemented by the memory 707, computing resources 706, software, instructions, programs, codes, etc. may include: any type of algorithm for artificial intelligence, machine learning, neural networks, etc., data movement, data management, data selection, filtering, encryption and / or decryption, compression and / or decompression, checksum calculation, hash value calculation, cyclic redundancy check (CRC), weight calculation, activation function calculation, training, inference, classification, regression, etc.

[0109] The computing device 702 or any other component disclosed herein may be implemented in any physical form factor. Examples of form factors may include 3.5 inches, 2.5 inches, 1.8 inches, etc., a storage device (e.g., a storage drive) form factor, an M.2 device form factor, an Enterprise and Data Center Standard Form Factor (EDSFF) (which may include, for example, E1.S, E1.L, E3.S, E3.L, E3.S 2T, E3.L 2T, etc.), an Add-In Card (AIC) (e.g., a PCIe card (e.g., a PCIe expansion card) form factor including half-height (HH), half-length (HL), half-height half-length (HHHL), etc.), a Next Generation Small Form Factor (NGSFF), an NF1 form factor, a Compact Flash (CF) form factor, a Secure Digital (SD) card form factor, a Personal Computer Memory Card International Association (PCMCIA) device form factor, etc., or a combination thereof. Any computing device disclosed herein may be connected to a system using one or more connectors, such as a SATA connector, a SCSI connector, a SAS connector, an M.2 connector, an EDSFF connector (e.g., a connector compatible with SFF-TA-1002 and / or SFF-TA-1009 (e.g., 1C, 2C, 4C, 4C+, etc.), a U.2 connector (which may also be referred to as an SSD Form Factor (SFF) SFF-8639 connector), a U.3 connector, a PCIe connector (e.g., a card edge connector), etc. In some embodiments, one or more of the computing devices 702 may be implemented as part of a SoC and / or a SIP.

[0110] Any computing device disclosed herein may be used in conjunction with one or more personal computers, smart phones, tablet computers, servers, server chassis, server racks, data rooms, data centers, edge data centers, mobile edge data centers, and / or any combination thereof.

[0111] In some embodiments, computing device 702 may be implemented with any device that may include or have access to memory, storage media, etc. for storing data that may be processed by one or more computing resources 706. Examples may include memory expansion and / or buffering devices (such as CXL type 2 and / or CXL type 3 devices, as well as CXL type 1 devices that may include memory, storage media, etc.).

[0112] Figure 8 An example embodiment of a computing system according to the disclosed example embodiments is shown. Figure 8 The computing system 800 shown in the figure can be used to implement any distribution scheme disclosed herein, including the distribution schemes described with respect to other figures, in which similar elements can be indicated by reference numerals ending with and / or containing the same numbers, letters, etc.

[0113] For illustrative purposes, Figure 8 The computing system 800 shown in can be described in the context of a system in which computing devices can communicate using an interconnect physical (PHY) layer (such as a PCIe physical layer) and one or more protocols (such as a PCIe protocol, CXL.cache, CXL.mem, CXL.io, etc.). However, aspects of the present disclosure can be implemented with computing devices using any other communication scheme, such as computing devices that can communicate using a network infrastructure (e.g., a network fabric) based on, for example, an Ethernet protocol.

[0114] The computing system 800 may include one or more CPUs 802a, one or more GPUs 802b, one or more computational storage devices (CSDs) 802c, one or more accelerators 802d, one or more memory expanders 802e, one or more memory devices 802f, one or more TPUs 802g, one or more NPUs 802h, one or more DSPs 802j, one or more FPGAs 802k, etc., which may be individually and / or collectively referred to as 802. One or more CPUs 802a, GPUs 802b, computational storage devices 802c, accelerators 802d, memory expanders 802e, memory devices 802f, TPUs 802g, NPUs 802h, DSPs 802j, and / or FPGAs 802k may respectively include computing resources 806a, 806b, 806c, 806d, 806e, 806f, 806g, 806h, 806j, and / or 806k, which may be individually and / or collectively referred to as 806.

[0115] Figure 8 Also shown is a host 801 that can be separated and / or integrated with the computing system 800. For example, in some embodiments, the host 801 can be implemented as a separate component that can run a computing workload (or workload) 810a that can "offload one or more computing tasks to one or more computing devices 802 in the computing system 800". In some other embodiments, the CPU 802a can be used as a host that can run a computing workload 810b that can "offload one or more computing tasks to one or more other computing devices 802". In such an embodiment, the CPU 802a (the CPU running the computing workload 810b or a different CPU) can also be used as a computing device. In other embodiments, the computing workload 810 can be partially run on a separate host 801 and partially run on one or more CPUs 802a, and one or more computing tasks can be offloaded to one or more other computing devices 802.

[0116] The computing system 800 may include allocation logic 804 that may implement any or all of the computing task allocation schemes disclosed herein or one or more portions thereof. Figure 8 As shown in , allocation logic 804 may be located at least partially at host 801. In some embodiments, some or all of allocation logic 804 may be located at multiple hosts, at one or more computing devices 802, at a user of computing system 800, and / or at any other location.

[0117] The computing device 802 may communicate using one or more communication connections 803a which, as mentioned above, may be implemented using a PCIe physical layer having one or more protocols (such as PCIe protocol, CXL.cache, CXL.mem, CXL.io, etc.) in some embodiments.

[0118] In embodiments using an interconnect physical layer (such as a PCIe PHY layer), one or more of the communication connections 803 may be implemented with one or more PCIe fabrics that may include one or more root complexes, switches, retimers, etc. For example, one or more communication connections 803b may be implemented with one or more root complexes at the CPU 802a and / or one or more switches that may enable the CPU 802a to communicate with any other computing device 802 and a communication interface 805 (e.g., a network interface card or controller, an interconnect card or controller, etc.) that may enable the computing system 800 to communicate with the host 801. In embodiments where the host 801 may be at least partially separate from the computing system 800, one or more communication connections 803a may be implemented with an interconnect (such as PCIe), a network (such as Ethernet), etc.

[0119] In some embodiments, the computing device 802f may be implemented with a memory module form factor, such as a dual in-line memory module (DIMM), that may implement one or more communication connections 803c with a memory interface, such as a double data rate (DDR) memory interface, a high bandwidth memory (HBM) interface, etc. In such embodiments, one or more computing resources 806f at the computing device 802f may be implemented, for example, with a processing in memory (PIM) function that may include computing resources on one or more memory dies, on one or more logic dies connected to one or more memory dies (e.g., stacked with one or more memory dies), etc.

[0120] Although computing system 800 is not limited to any particular physical configuration, in some embodiments, computing system 800 may be implemented with servers (such as compute servers, storage servers, etc.) configured as one or more chassis, blades, racks, clusters, data centers, edge data centers, etc.

[0121] The embodiments shown here are example operations and / or example components. In some embodiments, some operations and / or components may be omitted and / or other operations and / or components may be included. In addition, in some embodiments, the temporal order and / or spatial order of operations and / or components may be changed. Although some components and / or operations may be shown as separate components, in some embodiments, some components and / or operations shown separately may be integrated into a single component and / or operation, and / or some components and / or operations shown as a single component and / or operation may be implemented with multiple components and / or multiple operations.

[0122] Any functionality described herein (including any allocation logic, system management controller, etc.) may be implemented using hardware, software, firmware, or any combination thereof including, for example, hardware and / or software combinational logic, sequential logic, timers, counters, registers, state machines, volatile memory (such as DRAM and / or SRAM), non-volatile memory including flash memory, persistent memory (such as cross-grid non-volatile memory), memory with body resistance change, PCM, etc., and / or any combination thereof, CPLD, FPGA, ASIC, CPU that executes instructions stored in any type of memory (including CISC processors (such as x86 processors) and / or RISC processors (such as ARM processors), GPUs, NPUs, TPUs), etc. In some embodiments, one or more components may be implemented as one or more core particles (e.g., IC dies) in a SoC, a multi-chip module, a package (e.g., a SIP), etc.

[0123] Some embodiments disclosed above may be described in the context of various implementation details, but the principles of the present disclosure are not limited to various implementation details or any other specific details. For example, some functions have been described as being implemented by specific components, but in other embodiments, functions may be distributed between different systems and components in different locations and have various user interfaces. Specific embodiments have been described as having specific processing, operations, etc., but these terms also cover embodiments in which specific processing, operations, etc. can be implemented with multiple processing, operations, etc., or embodiments in which multiple processing, operations, etc. can be integrated into a single processing, step, etc. References to components or elements may represent only a portion of a component or element. For example, references to blocks may represent the entire block or one or more sub-blocks. Unless otherwise clear from the context, terms such as "first" and "second" used in the present disclosure and claims may only be used to distinguish the purpose of the elements they modify, and may not indicate any spatial order or temporal order. In some embodiments, references to elements may represent at least a portion of an element, for example, "based on" may represent "based at least in part on", etc. References to the first element may not imply the presence of a second element. The principles disclosed herein have independent utility and can be implemented separately, and not every embodiment can utilize every principle. However, the principles can also be implemented in various combinations, and some of the various combinations implemented can amplify the benefits of a single principle in a collaborative manner. The various details and embodiments described above can be combined to produce additional embodiments according to the inventive principles disclosed in this patent.

[0124] Since the inventive principles of this patent disclosure may be modified in arrangement and detail without departing from the inventive concept, such changes and modifications are considered to fall within the scope of the appended claims.

Claims

1. A method for assigning a machine learning task to a computing device, comprising: determining, by at least one processor, characteristics of a machine learning task; determining, by the at least one processor, a characteristic of a computing system, wherein the computing system comprises one or more computing devices; as well as The at least one processor allocates the machine learning task to at least one of the one or more computing devices based on characteristics of the machine learning task and characteristics of the computing system.

2. The method according to claim 1, wherein: The characteristics of the machine learning tasks include at least one of compatibility, priority, order, size, and type.

3. The method according to claim 1, wherein: The characteristics of the machine learning task include at least one of performance compatibility, efficiency compatibility, and latency compatibility.

4. The method according to claim 1, wherein: The characteristics of the computing system include at least one of a policy, a topology, a state, an operating parameter, and a scheduling algorithm.

5. The method according to claim 1, wherein: The characteristic of the computing system includes at least one of a performance policy and an efficiency policy.

6. The method according to claim 1, wherein: Characteristics of computing systems include policies, The characteristics of the machine learning task include: a first compatibility with a first computing device of the one or more computing devices based on the policy, and a second compatibility with a second computing device of the one or more computing devices based on the policy, and The allocating step includes allocating the machine learning task to a first computing device among the one or more computing devices based on the policy and the first compatibility.

7. The method according to claim 1, wherein: The characteristics of the computing system include a first strategy and a second strategy, The characteristics of the machine learning task include: a first compatibility with a first computing device among the one or more computing devices based on a first policy, and a second compatibility with a second computing device among the one or more computing devices based on a second policy, and The allocating step includes allocating the machine learning task to a first computing device among the one or more computing devices based on a first strategy and a first compatibility.

8. The method according to claim 1, wherein: The machine learning task is a first machine learning task, The characteristic of the machine learning task is a first characteristic of a first machine learning task, and The steps of allocation include: Selecting the first machine learning task based on the first characteristic of the first machine learning task, the second characteristic of the second machine learning task, and the scheduling algorithm; and Based on the selected step, a first machine learning task is assigned to at least one of the one or more computing devices.

9. The method according to claim 1, wherein: The machine learning task is a first machine learning task, and the method further includes: modifying operation of the first machine learning task on at least one of the one or more computing devices based on the priority of the first machine learning task and the priority of the second machine learning task; and Based on the modified step, a second machine learning task is assigned to at least one of the one or more computing devices.

10. The method according to any one of claims 1 to 9, further comprising: determining an operational state of at least one of the one or more computing devices; as well as Based on the operation status, the machine learning tasks are assigned to a data structure for storing the machine learning tasks.

11. The method according to claim 1, wherein: at least one of the one or more computing devices comprises a first computing device of the one or more computing devices, The characteristics of the machine learning task include: a first compatibility with a first computing device of the one or more computing devices, and a second compatibility with a second computing device of the one or more computing devices, and The method further comprises: determining an operational state of a first computing device among the one or more computing devices; and Based on the operating state and the second compatibility, the machine learning task is assigned to a second computing device among the one or more computing devices.

12. The method according to any one of claims 1 to 9, wherein: Characteristics of machine learning tasks include the size of the machine learning task, and The allocating step includes allocating the machine learning task to at least one of the one or more computing devices based on the size of the machine learning task.

13. The method according to any one of claims 1 to 9, wherein: At least one of the one or more computing devices includes a first computing device of the one or more computing devices, the method further comprising: modify characteristics of a computing system; and Based on the modified step, the machine learning task is assigned to a second computing device among the one or more computing devices.

14. The method according to claim 13, wherein: Characteristics of computing systems include policies.

15. The method according to claim 13, wherein: Characteristics of a computing system include operating parameters.

16. An electronic system comprising: at least one memory configured to store information for a machine learning task; A computing system comprising one or more computing devices; as well as At least one processor configured to: determining characteristics of the machine learning task based on the information; Determining characteristics of computing systems; and Based on characteristics of the machine learning task and characteristics of the computing system, the machine learning task is assigned to at least one of the one or more computing devices.

17. The electronic system of claim 16, wherein: Characteristics of computing systems include policies, The characteristics of the machine learning task include: a first compatibility with a first computing device among the one or more computing devices based on the policy, and a second compatibility with a second computing device among the one or more computing devices based on the policy, and The at least one processor is configured to assign a machine learning task to a first computing device among the one or more computing devices based on the policy and the first compatibility.

18. The electronic system of claim 16, wherein: The characteristics of the computing system include a first strategy and a second strategy, The characteristics of the machine learning task include: a first compatibility with a first computing device among the one or more computing devices based on a first policy, and a second compatibility with a second computing device among the one or more computing devices based on a second policy, and The at least one processor is configured to assign a machine learning task to a first computing device among the one or more computing devices based on a first policy and a first compatibility.

19. An electronic device comprising: at least one memory configured to store information for computing tasks; A computing system comprising one or more computing devices; as well as At least one processor configured to: determining characteristics of the computing task based on the information; Determine the characteristics of computing systems; as well as The computing task is allocated to at least one of the one or more computing devices based on characteristics of the computing task and characteristics of the computing system.

20. The electronic device according to claim 19, wherein: Characteristics of the computing task include performance compatibility with at least one of the one or more computing devices.

Images