Processing allocation in datacenter fleet

Abstract

The invention relates to process allocation in a datacenter fleet. A method and system for assigning tasks between processing devices of a data center. The method may include receiving a request to allocate a task to one or more processing devices, the request indicating a required bandwidth for executing the task, a list of predefined groups of processing devices connected to a host server, and indicating availability of the groups of processing devices included therein, the task assignment module is used for assigning tasks and available bandwidths to each available processing equipment group; the task is allocated to a group of processing devices whose available bandwidth is greater than or equal to the bandwidth required to execute the task, and the list is updated to indicate that each of the group of processing devices to which the task is assigned and other groups of processing devices sharing the at least one processing device is unavailable. Tasks may be assigned to a group of available processing devices having a minimum amount of power required.

Description

Description of the case This application is a divisional application of the patent application with application number 202011132249.X submitted to the China Patent Office on October 21, 2020, and the invention name is "Processing Distribution in Data Center Clusters". Technical Field

[0001] The present disclosure relates to processing distribution in a data center cluster. Background Art

[0002] Data centers typically contain fleets of servers with various processing devices, such as graphics processing units (GPUs) and accelerators connected to the servers. The processing devices can be assigned to various tasks, such as high-performance computing, cloud gaming, virtual machines, video encoding, infrastructure-as-a-service deployments, etc. The performance of a given task may involve processing at multiple processing devices, each of which is connected to a server. In addition to the server being able to act as a host and communicate with the processing devices, the processing devices may also be able to communicate peer-to-peer with each other, i.e., without relaying the communication through a server.

[0003] Data center server farms can contain hundreds or even thousands of servers with varying numbers of GPUs and accelerators connected to them. As a result, the bandwidth and power requirements of the server farm can be large. Therefore, it is necessary to carefully manage the bandwidth and power requirements of the server farm. Summary of the invention

[0004] The present disclosure provides methods and systems for power management of a server cluster in a data center while ensuring that sufficient bandwidth is provided for all tasks assigned to the server cluster. In order to provide these benefits, information about the processing device of each server connected to the server cluster can be obtained, and the information can be stored and accessed to determine which of the one or more processing devices to assign a task to. The candidate group of processing devices can be a non-exhaustive list. In summary, the obtained information is used to narrow the candidate list of all received tasks, further narrow the candidate list for any given task, and further select a single group from the candidates to perform the task. The obtained information can be updated periodically to reflect changes in server cluster attributes. The obtained information can also be updated periodically when tasks are assigned, tasks are completed, or both.

[0005] One aspect of the present disclosure is directed to a method, comprising: receiving, by one or more processors, a first request to assign a first task to one or more of a plurality of processing devices, each processing device being connected to a corresponding host server, at least some of the processing devices being configured to communicate with each other via a peer-to-peer connection, the first request indicating a required bandwidth for performing the first task; accessing, by the one or more processors, one or more lists of predefined processing device groups, each processing device group including one or more processing devices connected to a common corresponding host server, each list indicating the availability of the processing device groups of the corresponding server for the assigned task, and further indicating an available bandwidth for each available processing device group; assigning, by the one or more processors, the first task to the first processing device group, the available bandwidth of the first processing device group being greater than or equal to the bandwidth required for performing the first task; and updating, by the one or more processors, the list including the first processing device group to indicate that each of the first processing device group and any other processing device groups that share at least one processing device with the first processing device group are unavailable.

[0006] In some examples, updating the list may include indicating that the first processing device group is allocated and any other processing device groups that share at least one processing device with the first processing device group are unavailable.

[0007] In some examples, the method may further include receiving, by one or more processors, a second request to assign a second task to one or more of the plurality of processing devices, the second request indicating a required bandwidth for performing the second task; accessing, by the one or more processors, one or more lists including the updated list; assigning, by the one or more processors, the second task to a second group of processing devices having an available bandwidth greater than or equal to the required bandwidth for performing the second task, wherein the first group of processing devices and the second group of processing devices do not share any processing devices; and re-updating, by the one or more processors, the updated list to indicate that the second group of processing devices and each of any other groups of processing devices that share at least one processing device with the second group of processing devices are unavailable.

[0008] In some examples, the method may further include receiving, by one or more processors, an indication that the first task is completed; and updating, by the one or more processors, a list including the first processing device group to indicate that the first processing device group is available, and any other processing device groups that share at least one processing device with the first processing device group and do not share a processing device with another assigned processing device group are available.

[0009] In some examples, updating the list may include updating available bandwidth for one or more processing device groups that share at least one bus connection with the first processing device group. For a given processing device group for which available bandwidth is updated, the updated available bandwidth may be a difference between the available bandwidth before the update and the required bandwidth for executing the first task; or an amount of available bandwidth measured when executing the first task.

[0010] In some examples, the list of first processing device groups may include, for each available processing device group, further indicating an amount of additional power required to assign the first task to the available processing device group, and the method may further include designating the available processing group with the lowest amount of power required as the first processing device group. Updating the list of first processing device groups may include updating the amount of additional power required for one or more processing device groups that share at least one hardware component or mechanical component with the first processing device group. For a given processing device group for which the amount of additional power required is updated, the updated amount of additional power required may be determined based on a predetermined power requirement for the shared at least one hardware component or mechanical component.

[0011] In some examples, the first request may indicate each of a required host server bandwidth and a required peer bandwidth for performing the first task. The list including the first processing device group may indicate each of an available host server bandwidth and an available peer bandwidth for each available processing device group. Assigning the first task to the first processing device group may be further based on the available host server bandwidth of the first processing group being greater than or equal to the required host server bandwidth for performing the first task; and the available peer bandwidth of the first processing group being greater than or equal to the required peer bandwidth for performing the first task.

[0012] Another aspect of the present disclosure is directed to a method, the method comprising receiving, by one or more processors, a request to assign a task to one or more of a plurality of processing devices, each processing device being connected to a corresponding host server, at least some of the processing devices being configured to communicate with each other via a peer-to-peer connection, the request indicating a bandwidth required for performing the task, receiving, by the one or more processors, information for a plurality of processing device groups, each processing device group including one or more processing devices connected to a common corresponding host server, the information indicating, for each given processing device group, an available bandwidth for the given processing device group and an amount of additional power required to activate the given processing device group, selecting, by one of the plurality of processors, a processing device group having a lowest amount of additional power required for assigning the task from a subset of processing device groups having an available bandwidth greater than or equal to the bandwidth required for performing the task, and sending, by the one or more processors, a request to assign the task to the selected processing device group.

[0013] In some examples, the method may further include updating information for the plurality of processing device groups based on assigning the task to the selected processing device group. Updating the information may include updating, for each unselected group in the plurality of processing device groups, an amount of additional power required to activate the unselected processing device group.

[0014] In some examples, the request may further indicate a group size parameter, which may indicate, for each given processing device group, a group size of the given processing device group, and the processing device group having the lowest amount of additional power required for assigning the task may be selected from a subset of processing device groups having an available bandwidth greater than or equal to a bandwidth required for performing the task and having a group size within the group size parameter. The group size of the given processing device group may indicate a total number of accelerators, graphics processing units, or both included in the given processing device group.

[0015] In some examples, the request to assign the task to the selected group of processing devices may include sending, by one or more processors, an instruction to initiate the task to a host server of the selected group of processing devices.

[0016] Another aspect of the present disclosure is directed to a method, including storing, by one or more processors of a server, information about a plurality of processing devices connected to and controlled by the server, wherein at least some of the plurality of processing devices are configured to communicate with each other via a peer-to-peer connection, wherein the stored information includes identifications of subgroups of the plurality of processing devices, each identified subgroup being associated in the stored information with a corresponding availability indication, a corresponding available bandwidth indication, and a corresponding activation required power indication, receiving, by the one or more processors of the server, a request to assign a task to a specified subgroup, assigning, by the one or more processors of the server, the task to the specified subgroup, and in response to assigning the task to the specified subgroup, updating, by the one or more processors of the server, the stored information to indicate changes in availability, available bandwidth, and power required for activation of at least some of the plurality of processing devices.

[0017] The method may further include receiving, by one or more processors of the server, a request for the stored information from a requesting node; and sending, by one or more processors of the server, the stored information to the requesting node in response to the request. The request to assign a task may be received from the requesting node.

[0018] In some examples, the method may further include collecting information about the plurality of processing devices, wherein collecting information about a non-exhaustive list of subgroups of the plurality of processing devices, wherein the subgroups are excluded from the non-exhaustive list based at least in part on the number of processing devices included in the excluded subgroups. Storing the information about the plurality of processing devices may involve storing compiled information. The non-exhaustive list of subgroups includes a maximum number of subgroups for each number of processing devices up to a predetermined maximum number, and the method may further include: for each subgroup of processing devices within the predetermined maximum number, determining a number of links connecting the processing devices of the subgroup, and for each number of processing devices up to the predetermined maximum number, selecting at most a maximum number of subgroups with a minimum number of links.

[0019] Another aspect of the present disclosure is directed to a system comprising a data center comprising a plurality of host servers. Each host server may be connected to a separate group of processing devices. At least some of the processing devices may be configured to communicate with each other via a peer-to-peer connection. Each host server includes packet data identifying a subgroup of the plurality of processing devices, each identified subgroup being associated in the packet data with a corresponding availability indication, a corresponding available bandwidth indication, and a corresponding power indication required for activation. The system may further include a memory for storing instructions. The system may further include one or more processors coupled to the memory and configured to execute the stored instructions.

[0020] Executing the instructions may involve performing any of the methods described in any one or combination of the embodiments herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram of an exemplary data center according to aspects of the present disclosure.

[0022] Figure 2 is a block diagram of an exemplary server according to aspects of the present disclosure.

[0023] Figure 3 is a flow chart representing an exemplary task allocation routine according to aspects of the present disclosure.

[0024] Figure 4-6 According to aspects of the present disclosure Figure 3 Flowchart of an exemplary subroutine of a task allocation routine.

[0025] Figure 7 is a flow chart of an exemplary information collection and storage routine according to aspects of the present disclosure. DETAILED DESCRIPTION Overview

[0026] The technology of this disclosure generally relates to efficiently distributing tasks among a network of processing devices in order to optimize the power requirements of the network for performing the tasks.

[0027] To manage the bandwidth and power requirements of the server farm, each host may store a grouping table that lists the different groups of processing devices connected to the host. The grouping table may include information related to various attributes of the processing device groups, including, but not limited to: the number of processing devices included in the group (group size); ID numbers of the grouped processing devices (grouped devices); the total number of links required to connect the processing devices of the group to the host (links to host); the total number of links required to connect the processing devices to each other via peer-to-peer connections (P2P links); the total number of links required to connect the processing devices to the host and to each other via peer-to-peer connections (total links); the available bandwidth for communication between the host and the processing devices in the group (host-to-device bandwidth); the available bandwidth for communication between the processing devices via peer-to-peer connections (P2P bandwidth); the amount of power required to activate the group (power to activate); and the current state or availability of the group (state).

[0028] In some implementations, the system may receive tasks for assignment, such as from one or more users or applications. Each received task may specify bandwidth requirements, such as host-to-processing device bandwidth, peer-to-peer bandwidth, or both.

[0029] In some embodiments, task allocation may involve determining which groups have available bandwidth that meets or exceeds the required bandwidth specified by the task, and then selecting from those groups the group with the lowest amount of power required to activate the group. In this regard, the power consumed in allocating a given incoming task is minimized while providing sufficient processing resources and communication bandwidth to effectively complete the task.

[0030] In some embodiments, each received task may further specify a processing device group size on which the task should be allocated, such as the number of processing devices or a minimum number of processing devices. In such embodiments, task allocation may involve determining which groups have group sizes within the task requirements and selecting from among those groups having the specified group sizes.

[0031] In addition to the task allocation routines and processes, the system may also perform a group table management process. The group table management process may include initially constructing and populating a group table. This may be performed by each host of the server farm for the corresponding processing device to which it is connected. A graph traversal algorithm may be used to identify links between processing devices, groups may be identified based on the links, and a limited number of groups may be selected to populate the group table so as to minimize the number of links for each group in the group table. If the network of processing devices is a tree network, a tree traversal algorithm may be used. For a point-to-point topology, such as a peripheral component interconnect express (PCIe), a breadth-first search algorithm may be used.

[0032] Group table management may further involve updating a group table of a host as tasks are assigned to processing devices of the host. Updating the group table may involve indicating the unavailability of a processing device when a task is assigned to the processing device, or indicating the unavailability of those groups when tasks are assigned to processing devices of those groups. Similarly, a processing device or group may be updated to indicate availability after completing an assigned task. Updating the group table may further involve indicating the remaining available bandwidth for each group listed as tasks are assigned to processing devices of the network. For example, if two groups share a link in the network, assigning a task to one group may result in a reduction in the bandwidth available to the other group on the shared link. Updating the group table may further involve indicating the updated power required to activate other groups of the network. For example, if two groups share a hardware component or a mechanical component, activating one group may reduce the power requirement for activating the second group because the hardware or mechanical component has been powered on and may already be running due to the activation of the first group.

[0033] The above-described embodiments can reduce the cost of assigning tasks among processing nodes, particularly for large server farms, such as those included in a data center. The reduced cost can include any one or combination of the following: reduced processing power by selecting only the required processing devices, reduced power consumption by using as little power as possible to activate the hardware and mechanical components of the data center to complete the task, and reduced time to complete the assigned task by avoiding assigning tasks to groups with insufficient resources or bandwidth. Exemplary Systems

[0034] Figure 1 An exemplary system 100 is shown including a data center server farm 110. A plurality of server racks 120 may be included in the server farm 110. Each server rack 120 may further include a plurality of server trays 125, and each server tray 125 may include a corresponding server 130. Figure 2 An exemplary server 130 is described in further detail.

[0035] The data center server farms may implement any of a variety of architectures and technologies, including but not limited to direct attached storage (DAS), network attached storage (NAS), storage area network (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), hybrid architecture networks, etc. The data center may include many other devices in addition to the processing and storage devices of the servers 130, such as cables, routers, etc. Furthermore, in some examples, the data center may be a virtualized environment.

[0036] Each server 130 included in the system can host multiple processing devices P1-P8, such as one or more graphics processing units (GPUs), one or more accelerators, etc. For illustrative purposes, only the processing device P1 is generally shown, and it is not specified which device is a GPU, which device is an accelerator, etc. The processing devices P1-P8 are connected to the server 130 or to each other via multiple switches S1-S6. The processing devices and switches can be arranged according to a PCIe topology, whereby the switches can be PCIe switches. In an alternative, other tree bus topologies can be utilized. The server farm 110 can include hundreds - even thousands - of servers, each server having a varying number of processing devices connected thereto via a bus topology such as PCIe.

[0037] In some examples, the topology of servers and processing devices may be a tree topology, whereby each processing device is connected to a server through a series of one or more switches. In other examples, the topology may include additional bus links such that a given processing device is connected to a server through more than one series of switches. For example, in Figure 1 In the example, each processing device P1, P2, P4, P5, P6, P7 and P8 is connected to the server 130 through a series of switches (P1 to S1; P2 to S2, S1; P4 to S4, S3, S1; P5 to S3, S1; P6 to S5; P7 to S6, S5; and P8 to S6, S5), and the processing device P3 is connected to the server through multiple series of switches (through S2, S1; or through S4, S3, S1).

[0038] Figure 2 is an exemplary server 200 - such as Figure 1 The server 200 may include a processor 210, a memory 220, and other components that are typically present in a server.

[0039] Processor 210 may be a well-known processor or other less well-known type of processor. Alternatively, processor 210 may be a dedicated controller, such as an ASIC.

[0040] The memory 220 may store information accessible by the processor 210, including data 230 that may be retrieved, manipulated, or stored by the processor 210. The memory 220 may also include instructions 240 that may be executed by the processor 210. The memory 220 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 210, such as a hard drive, a solid-state drive, a tape drive, an optical storage, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a write-only memory, and a read-only memory.

[0041] Data 230 may be retrieved, stored, or modified by processor 210 according to instructions 240. For example, although the systems and methods are not limited to a particular data structure, data 230 may be stored in a computer register, in a database, as a structure with multiple different fields and records, in a file, or in a buffer. Data 230 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 230 may include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information used by a function to calculate related data.

[0042] The instructions 240 may be a set of instructions directly executed by the processor 210, such as machine code, or an instruction set indirectly executed, such as a script. In this regard, the terms "instructions," "steps," and "programs" are used interchangeably herein. The instructions 240 may be stored in an object code format for direct processing by the processor 210, or in other types of computer languages, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled.

[0043] although Figure 2 The processor 210 and memory 220 are functionally shown as being within the same block, but the processor 210 and memory 220 may actually include multiple processors and memories that may or may not be stored within the same physical housing. For example, some data 230 and instructions 240 may be stored on a removable CD-ROM, while other data 230 and instructions 240 may be stored within a read-only computer chip. Some or all instructions and data may be stored in a location physically remote from the processor 210 but still accessible to the processor 210. Similarly, the processor 210 may actually include a collection of processors that may or may not operate in parallel.

[0044] The server 200 may include a number of other devices in addition to the processor and storage device, such as a communication device 250 to enable input and output between computing devices, such as cables, routers, etc. The communication device 250 may facilitate communication between the server 200 and user devices or other remote devices that communicate with the data center, and may be able to transmit data to and from other computers such as modems (e.g., dial-up, cable, or fiber optic) and wireless interfaces. For example, the server 200 and other servers and processors in the data center may receive communications via a network connection, such as through the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-specific communication protocols, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n or other such standards), as well as RPC, HTTP, and various combinations of the foregoing.

[0045] The memory 220 of each computing device may store information accessible by the one or more processors 210 , including data 230 received at or generated by the computing device 200 , and instructions 240 executable by the one or more processors 210 .

[0046] Data 230 may include a grouping table 232. Grouping table 232 may list multiple processing device groups and may further indicate various attributes of the processing devices. Figure 1 An exemplary grouping table of processing device groups included in the system 100 is shown in FIG. Table 1:

[0047] In the example of Table 1, the grouping table includes 7 processing device groups with a group size of 1, 7 processing device groups with a group size of 2, and 7 processing device groups with a group size of 3. The group size of each group corresponds to the total number of processing devices included in the group.

[0048] In some examples, the groups included in the grouping table may not be an exhaustive list of all groups. For example, the number of groups listed for each given group size may be limited by a predetermined value. The predetermined value may be one of the grouping table parameters 234 stored in memory and used to control the population of the grouping table 232. For example, Figure 1 The server 130 of the system 100 is connected to eight processing devices P1 to P8, but the grouping table of the server 130 may be limited to seven groups. In such an example, one of the processing devices may be excluded from a group with a group size of 1, as shown in Table 1.

[0049] The determination as to which processing device to exclude may be based on predefined rules also included in the grouping table parameters 234. For example, the parameters may specify one or more rules by which certain candidate groups are prioritized over other candidate groups based on one or more attributes of the respective groups. Such attributes may include, but are not limited to, the various group attributes shown in Table 1, including the total number of links required to connect the processing devices of the group to the host (Links to Host), the total number of links required to connect the processing devices to each other via peer-to-peer connections (P2P Links), the total number of links required to connect the processing devices to the host and to each other via peer-to-peer connections (Total Links), the available bandwidth for communication between the host and the processing devices in the group (Host-to-Device Bandwidth), the available bandwidth for communication between the processing devices via peer-to-peer connections (P2P Bandwidth), and the amount of power required to activate the group (Power for Activation). Since the population of the grouping table occurs before tasks are assigned to any processing device, the current state or availability of each group (State) is not a factor in selecting a candidate group, as each group must start as "Available".

[0050] In the example of Table 1, total links are used to select candidate groups, whereby groups with fewer total links are preferred over groups with more total links. In the example of group size 1, P4 is excluded because it is the only processing device with 4 total links, while every other processing device has 3 or fewer total links.

[0051] In the example of group size 2, only pairs of processing devices with 4 or fewer links are included in the grouping table. For example, group P1, P5 is included because processing devices P1 and P5 are linked to server 130 and to each other via no more than 4 links: P1-S1, P5-S3, S3-S1, and S1-server. In contrast, group P2, P5 is excluded because at least 5 links are required to link processing devices P2 and P5 to each other and to server 130: P2-S2, S2-S1, P5-S3, S3-S1, and S1-server.

[0052] In the example of a group size of 3, only groups of processing devices having 6 or fewer links are included in the grouping table. For example, group P1, P2, P5 are included because processing devices P1, P2, and P5 are linked to server 130 and to each other via no more than 6 links: P1-S1, P2-S2, S2-S1, P5-S3, S3-S1, and S1-server. In contrast, group P2, P3, P4 are excluded because at least 7 links: P2-S2, S2-S1, P3-S2 (or P3-S4), P4-S4, S4-S3, S3-S1, and S1-server are required to link processing devices P2, P3, and P4 to each other and to server 130.

[0053] In some cases, two processing groups may have the same attributes for determining which group is included in the packet table, but the packet table may be restricted so that only one group can be selected. In this case, the group may be selected randomly or according to a secondary rule. For example, for the group size 3 of Table 1, groups P1, P3, P5 are not included in the packet table, although P1, P3, and P5 are linked to the server 130 and are linked to each other by no more than 6 links: P1-S1, P3-S2, S2-S1, P5-S3, S3-S1, and S1-server. Other candidates on P1, P3, and P5 may be random. Alternatively, other factors may have been considered when excluding this group in favor of other groups.

[0054] A similar determination may be made for each packet size, thereby selecting for inclusion in the packet table 232 those groups of processing devices for a given packet size that have the fewest total links (or, in a different example, some other property that is best).

[0055] return Figure 2 In some examples, the grouping table parameters 234 may further include a group size threshold indicating a maximum group size, a minimum group size, or both. In the example of the grouping table of Table 1, the grouping table includes processing device groups with group sizes of 1, 2, and 3, but other processing device groups with larger group sizes (e.g., 4, 5) are also available. This may be because the grouping table parameters limit the grouping table to a group size of 3. For further example, since the grouping table of Table 1 includes a group with a group size of 1, it can be seen that the grouping table has no minimum group size.

[0056] Instructions 240 may include various algorithms for instructing processor 210 to perform operations according to the routines described herein. For example, instructions 240 may include a grouping table assembly routine 242 for populating a grouping table, such as according to the exemplary rules described above. For further examples, instructions may include a task allocation routine 244 for determining which processing device groups an incoming task should be assigned to. For further examples, instructions 240 may include a grouping table management or update routine for updating the attributes of the processing device groups included in the grouping table. In conjunction with Figure 3-7 , describing these routines and other routines in more detail. It should be understood that the routines described herein are only examples, and in other examples, certain steps may be added, subtracted, replaced or reordered. Exemplary Methods

[0057] Figure 3 is a flow chart illustrating an exemplary routine 300 for allocating tasks received at a data center. The routine may be executed by one or more processors of a task allocator included in the data center. Figure 2 In the example of the present disclosure, the data and instructions of the task distributor are shown as being included in the server of the data center. However, in other examples of the present disclosure, the task distributor may include one or more processors and one or more memory storage devices separate from the server of the data center, or may include some combination of server processors and memories and separate processors and memories.

[0058] At block 310, one or more processors may receive a request to assign a task to one or more processing devices of a data center. The request may be received from a user device or other remote device in communication with the data center and capable of transmitting data to and from other computers.

[0059] The received request may include instructions for performing a processing operation associated with the task. In some examples, the received request may further include an indication of the amount of bandwidth required to perform the task. In some examples, the received request may further include an indication of the number of processing devices required to perform the task. Since not all processing devices included in a data center necessarily have sufficient bandwidth to perform a task, and not all groups of processing devices included in a data center necessarily include the required number of devices, this information can be used by one or more processors to determine candidate groups of processing devices for performing the task, and more specifically, to exclude groups of processing devices that do not meet the above criteria for performing the task.

[0060] In box 320, one or more processors may access a list of predefined processor groups. The list may be stored in a server included in the data center and may be accessed from the server. In some examples, each server included in the data center may store a separate list including processing devices connected to the server. In this way, the processing devices of the servers connected to the data center may be included in multiple separately stored lists. In such an example, box 320 may involve accessing the list from each server in the data center, or more specifically, accessing the list from each server including the list. One or more processors may be configured to aggregate the accessed lists in order to process information related to all processing devices connected to the data center servers. For clarity, the examples below refer to "lists", but it should be understood that the same or similar principles may be applied to multiple lists accessed and evaluated by one or more processors.

[0061] In some examples, the list can be stored in the form of a table, although the list can be stored in another structure. The list can include the above combined Figure 2 The information shown in Table 1 is shown. That is, for each processing device group, the list may include any one or combination of the following attributes:

[0062] Group Size: The number of processing devices included in the group.

[0063] Grouped devices: ID numbers of the processing devices included in the group.

[0064] Links to Hosts: The total number of links used to connect the processing devices included in the group to the connected servers.

[0065] P2P links: the total number of links used to connect processing devices included in a group to each other via peer-to-peer connections.

[0066] Total Links: The total number of links used to connect the processing devices included in the group to the connected servers and to each other. In many cases, two processing devices included in the group may share one or more links connected to a server, or the link that links one processing device to another may also be the link that links one of the processing devices to a server. In this case, the "Total Links" may be less than the sum of the values ​​of the Link to the Host and the P2P Link.

[0067] Host-to-Device Bandwidth: The available bandwidth for communication between connected servers and the processing devices included in the group. In some cases, the available bandwidth in the link connecting the server to one processing device may be different from the available bandwidth for connecting to another processing device. In this case, the value listed here is the minimum value among all the processing devices included in the group, because the assignment of tasks to the group will be limited to the processing device with the lowest available bandwidth.

[0068] P2P Bandwidth: The available bandwidth for communication between the processing devices included in the group via peer-to-peer connections. As with host-to-device bandwidth, the available bandwidth in the link connecting one processing device to another processing device can vary from one peer-to-peer connection to the next. In this case, the value listed here is the minimum value among all the peer-to-peer connections of the processing devices included in the group, because the assignment of tasks to the group will be limited to the peer connection with the lowest available bandwidth. The available bandwidth may change as tasks are assigned to or completed by processing devices connected to servers, thereby exhausting the available bandwidth and leaving less available bandwidth for future tasks.

[0069] Power for activation: The amount of power required to activate the processing devices included in the group if a task is to be assigned to the group. In some examples, power may be required to operate hardware components of the processing devices of the group, such as powering application specific integrated circuits (ASICs), GPUs, trays, and processor boards. In some examples, power may be required to operate mechanical components associated with the processing devices of the group, such as powering server fans or cooling equipment to maintain the temperature of the server cluster.

[0070] Status: The current state or availability of a processing device group. The status may indicate when the group is available or unavailable for assignment of tasks. The status may further differentiate between a group that is unavailable because a task has been assigned to the group and a group that is unavailable because one or more processing devices included in the group are also included in a different group to which a task is assigned.

[0071] In box 330, one or more processors can assign tasks to a group of processing devices included in the group. In some examples, assigning tasks can involve first identifying a candidate group from a list according to a first set of criteria, and then selecting one of the candidate groups from the list according to a second set of criteria. The first standard can filter out the processing device group to which tasks cannot be assigned, such as a group that does not have enough number of processing devices to perform tasks or does not have enough bandwidth to perform tasks. Insufficient bandwidth can refer to insufficient bandwidth between a server and each processing device, insufficient bandwidth in a peer-to-peer connection between processing devices, or a combination of the two. The first standard can also be used to filter out unavailable processing device groups, because another task has been assigned to the group, or because at least one processing device included in the group is included in different groups to which different tasks have been assigned and have not yet been completed. Based on factors such as the lowest or highest available bandwidth, the lowest required power, the lowest or highest number of processing devices, the second standard can be used to select the best group from the candidate group.

[0072] Figure 4 It is shown Figure 3 330 of the example subroutine 400, which assigns tasks to a group of processing devices. The operation of the routine 300 may start at block 320 to continue Figure 4 The operation of block 410 will be combined Figure 1 An example of the subroutine 400 is described with reference to the system 100 shown and the system properties shown in Table 1. In this example, the received task requires a group size of at least 2 processing devices, a host-to-device bandwidth of 5 Gbps, and a peer-to-peer bandwidth of 2 Gbps.

[0073] At block 410, one or more processors may determine which processor groups are within the specified group size parameters. For example, one or more processors may exclude the first seven processor device groups of Table 1 because the received tasks require a group size of at least 2 processing devices and each of these groups includes only one processing device.

[0074] At block 420, the one or more processors may determine which processing device groups have available bandwidth greater than or equal to the required bandwidth for executing the task. Continuing with the above example, the one or more processors may exclude groups (P1, P5), (P1, P4, P5), (P1, P2, P5), (P2, P3, P5), and (P3, P4, P5) that have insufficient host-to-device bandwidth. The remaining processing device groups constitute a subset of the list of processing device groups, which itself is a list of candidate processing device groups to which the task may be assigned.

[0075] At block 430, the one or more processors may select, from the subset of the groups of processing devices, the group of processing devices that has the lowest amount of additional power required to perform the task. Continuing with the example above, the one or more processors may select (P2, P3) or (P6, P7) from the list because each of these groups requires 220 W of additional power to perform the task, while the other groups require more than 220 W. The selection between these two groups may be random or based on other criteria, such as which group has a higher available bandwidth. The result of selecting one of these two groups is a reduction in the power consumed to perform the task compared to the power consumed by the groups assigned using other known processes. Over time, the power savings resulting from task assignments such as those described above may be as high as several kilowatt-hours per day.

[0076] At block 440, the one or more processors assign the task to the selected set of processing devices. Assigning the task may involve providing the task to a server connected to the set of processing devices and providing instructions to the server to perform the task, including specifying a processing device to be used to perform the task.

[0077] return Figure 3 After the tasks are assigned, operations may continue at block 340 where one or more processors update the list. In some examples, updating the list may involve updating an indication of the availability of the processor group included in the list. In other examples, other attributes included in the list may also be updated, such as "bandwidth" and "power to activate" parameters.

[0078] Figure 5 It is shown for Figure 3 340 of the exemplary subroutine 500, which updates the list of processing device groups. The operation of the routine 300 may start at block 330 to continue Figure 5 The operation of block 510 in the above Figure 3 and 4 The examples provided here continue to further illustrate the operation of the system.

[0079] In block 510, one or more processors may update the status of the processor device groups included in the list. In addition, the status of other groups including P2 processing devices or P3 processing devices may be updated to "unavailable", which means that the group is not available for the assigned task because at least some of its processing devices are occupied by another task.

[0080] Figure 6 It is shown Figure 5 FIG. 5 is a flowchart of an exemplary subroutine 600 of block 510 of FIG. 5 , which updates the status of the processor device groups in the list. In block 610, one or more processors may change the status of the selected processing device group from "available" to "assigned". Continuing with the above example, assuming that the task is assigned to group (P2, P3), the status of group (P2, P3) may be updated from "available" to "assigned", which means that the task has been assigned to the group and cannot be assigned another task. In other examples, the status may be changed to "unavailable", which means that the group is not available for the assigned task. The "unavailable" status may be more general than the "assigned" status because it indicates which groups are unavailable, but does not indicate that the task has been assigned to the group.

[0081] At block 620, the one or more processors may determine which other processing device groups (i.e., other than the processing device group to which the task was assigned at block 330) share at least one processing device with the selected processing device group. Continuing with the above example, since processing devices P2 and P3 are included in group (P2, P3), the other groups included in the determination would include (P2), (P3), (P1, P2), (P1, P3), (P1, P2, P3), (P1, P3, P4), (P1, P2, P5), (P2, P3, P5), and (P3, P4, P5). It should be noted that the determination of block 620 applies not only to a subset of the candidate processing device groups, but to all processing device groups included in the list.

[0082] At block 630, one or more processors may change the state of the processing device group determined at block 620 to share at least one processing device with the selected processing device group. The state may be changed from "available" to "unavailable", indicating that at least one processing device included in each group is unavailable for other tasks to be assigned.

[0083] return Figure 5 At block 520, one or more processors may update the "bandwidth" attribute of the processing device groups included in the list based on the assignment of the task. Continuing with the above example, assuming that the task is assigned to group (P2, P3), this may reduce the available bandwidth of the link between server 130 and processing devices P2 and P3, such as Figure 1As shown, these links are links P2-S2, P3-S2, S2-S1 and S1-server. As a result, other processing devices that rely on one or more of these links to connect to the server or to other processing devices via peer links have less available bandwidth left after assignment of tasks.

[0084] In some examples, updating the "bandwidth" attribute can be accomplished through monitoring. For example, available bandwidth can be obtained based on actual link utilization monitoring. The monitoring function can be performed and controlled by one or more processors of a task allocator, a data center server, or some combination thereof.

[0085] In some examples, updating the "bandwidth" attribute can be accomplished by estimation. For example, the amount of bandwidth required by the task can be subtracted from the amount of available bandwidth in the group to which the task is assigned in order to derive the updated amount of available bandwidth. Continuing with the example above, since the required host-to-device bandwidth for the task is 5 Gbps and the required P2P bandwidth is 2 Gbps, these values ​​can be subtracted from the available bandwidth of the assigned groups (P2, P3). Additionally, other groups that share a link with the assigned group can be estimated to have a similar drop in available bandwidth. Again, continuing with the example above, for groups that include link S1-Server - which is all groups that include P1, P2, P3, P4, or P5, the bandwidth may be reduced by 5 Gbps, while for P6, P7, and P8, the bandwidth may remain unchanged.

[0086] It should be noted that in some cases, assigning a task to a group of processing devices may affect the bandwidth of other groups, even if those groups do not share processing devices with the assigned group. For example, as described above for group (P2, P3), assigning a task to that group may affect the bandwidth of other groups that share the S1-server link. In a similar manner, assigning a task to group (P1, P5) may reduce the bandwidth on link S3-S1 that is also available to processing device group (P3, P4), even though that group does not share any processing devices with group (P1, P5).

[0087] Furthermore, measurement and estimation techniques are not mutually exclusive, as some systems may rely on regular estimates and periodic corrections of the estimates based on measurements.

[0088] At block 530, one or more processors may update a "power to activate" attribute for a processing device group included in the list. Activating a processing device group may involve activating a hardware or mechanical component that was previously inactive, idle, or in a low-power state. The hardware component may include any one or combination of an ASIC, GPU, tray, or board of a processing device included in the assigned processing device group. The mechanical component may include any one or combination of a server fan or a cooling device for a processing device included in the assigned processing device group. Other hardware or mechanical components may also be included. Since a given hardware component or a given mechanical component may provide services for more than one processing device, activating the hardware or mechanical component to provide services for one processing device may mean that the component has been activated and does not need to be activated again to provide services for a second processing device such as a device included in a different processing device group. In this way, activating hardware and mechanical components for the purpose of activating one processing device group may result in less power required to activate hardware and mechanical components of another processing device group because some components may have already been activated.

[0089] Continuing with the above example of Table 1, the selected processing device group (P2, P3) requires 220W to activate. Since all hardware and mechanical components are activated when a task is assigned, the power required for activation may drop to 0W. In addition, for other processing device groups, such as those that share processing device P2 or P3, the power required for those groups may also drop because the mechanical and hardware components of those processing devices have been activated. In addition, other processing devices may share hardware or mechanical components with P2 or P3, so activation of the processing device groups including those processing devices may also be reduced. For example, if P1 and P2 share hardware components, and if P3, P5, and P7 also share mechanical components, then for all processing device groups that include one of P1, P5, or P7, the power required for activation may drop, even if none of those processing devices are currently assigned a task.

[0090] To illustrate an example of one or more processors updating the attributes of a processing device group after assigning a task to one of the processing device groups, Table 2 shows the attributes of the one or more processors for updating the attributes of the processing device group. Figure 1 The system 100 includes an updated grouping table for processing device groups: Table 2:

[0091] In the example of Table 2, the bandwidth, power for activation, and state attributes of many processing device groups are updated to reflect: the reduction in available bandwidth due to the assignment of the first task; the reduction in power required to activate other processing device groups due to the components of the (P2, P3) group having been activated; and the unavailability of some processing groups due to the processing devices P2 and P3 being in use.

[0092] Despite Figure 3 , but after completing the task, one or more processors may further update the properties of the processing device group again to reflect the availability of processing devices P2 and P3 and that the processing device group (P2, P3) is no longer assigned a task. This may involve the properties shown in Table 2 reverting to those shown in Table 1, provided that another task has not yet been assigned to any of the other processing devices.

[0093] In another case, before completing the first task, one or more processors may receive another task for assignment. In this case, the one or more processors may rely on the properties shown in Table 2 to effectively assign the next task to a group of processing devices that still have sufficient available bandwidth and for which the least amount of power is required to activate the processing devices given that the first task has caused some hardware and mechanical components to be activated.

[0094] To illustrate, continuing with the above example, if the second received task requires a group size of 3 or greater, 1 Gbps of host-to-device bandwidth, and 1 Gbps of P2P bandwidth, the one or more processors may determine a subset of candidate processing device groups that excludes all groups with a group size of 1 or 2 and the "unavailable" (P1, P2, P3), (P1, P3, P4), (P1, P2, P5), (P2, P3, P5), and (P3, P4, P5). This would leave two candidate processing device groups: (P1, P2, P5) and (P6, P7, P8). The one or more processors may then assign the task to (P6, P7, P8) based on the power required to activate the group being 440 W, which is less than the power required to activate (P1, P4, P5). The one or more processors may then further update the properties of the processing device groups to reflect that (P6, P7, P8) are assigned tasks, that other groups including at least one of P6, P7, or P8 are unavailable, reduced available bandwidth for at least some of the other processing device groups, and the minimum power required to activate at least some of the other processing device groups.

[0095] In some examples, the one or more processors may also be configured to populate the list of processing device groups and their attributes in the form of a table or in some other form or data structure prior to receiving the first task. Figure 7 is a flow chart illustrating an exemplary routine 700 for collecting and storing information about one or more processing device groups connected to a server. Figure 3 Prior to block 310 of routine 700, the steps of routine 700 are performed in whole or at least in part.

[0096] In box 710, one or more processors may collect information about processing devices connected to the server and controlled by the server. The information may include data about the connection between the processing devices connected to the server, such as which devices are peer-to-peer connected, through which links, and through which switches. Although not shown in the above example for simplicity, the information included in the list may further include specific links for connecting the server, the processing device, and the switch to each other. One or more traversal algorithms, such as a tree traversal algorithm, or more specifically, such as a breadth-first search algorithm, may be used to collect this information. The information collected from the traversal algorithm can be used to determine the total number of links connecting each processing device group. The information may also include measurements of the bandwidth between the server and each processing device, measurements between each processing device via peer-to-peer connections, measurements of the power of hardware and mechanical components associated with each device, and the like.

[0097] At box 720, one or more processors may determine a non-exhaustive list of processing device subgroups. As previously described, the non-exhaustive list may be limited based on a predetermined value such as a group size parameter, which limits the group size of any given subgroup of processing devices. For example, in the above examples of Tables 1 and 2, the non-exhaustive list represented by those tables includes two restrictions: the group size is limited to 3, i.e., no more than three processing devices in a subgroup, and a restriction to 7 subgroups for each group size. In other examples, different parameters may be selected. For example, the number of subgroups for each group size may be limited by the total number of processing devices connected to the server, thereby not limiting groups with a group size of 1. In such an example, Tables 1 and 2 will include 24 entries: 8 groups of size 1, 8 groups of size 2, and 8 groups of size 3.

[0098] The selection of subgroups to be included in the non-exhaustive list may be based on other factors and parameters, such as the total number of links included in each group. In the examples of Tables 1 and 2, the subgroups are selected to minimize the total number of links, so that subgroups with more total links are excluded from the list in favor of subgroups with fewer total links. In other examples, other factors, such as bandwidth, may be considered. In some cases, the exclusion of subgroups from the list may be affected by the inclusion of other subgroups, such as whether each processing device is limited to appearing in the list a maximum number of times.

[0099] At block 730, information about the subsets included in the non-exhaustive list is stored. For example, each list may be stored on a respective server connected to the processing devices included in the list.

[0100] exist Figure 7 In the example, the routine Figure 3 However, in some cases, it can be executed at a later time. Figure 7 For example, one or more processors may periodically monitor all bus links connected to the server and update the information included in the list based on new measurements. In this regard, the available bandwidth for some processing device groups may change based on updated collected information.

[0101] In some examples, the operations of routines 300 and 700 may be performed by separate processors. For example, one or more processors of a server connected to the processing devices included in the list may be responsible for performing the operations of routine 700, because these operations are limited to collecting information from the devices connected to a specific server, while one or more processors of a task distributor connected to many servers of the system may be responsible for performing the operations of routine 300, because when a task is received, the list from several servers may be queried in order to find the best or most efficient group of processing devices for distributing the task.

[0102] In this regard, it should be recognized that the above Figure 3-6 The examples described generally describe a simplified example of assigning tasks to processing devices associated with a single server based on a single list and updating the single list after the tasks are assigned, while Figure 7 A simplified example of populating a single grouping table for a single server is generally described. However, in other examples, many servers may include corresponding lists of processing device groups, and one or more processors of the job allocator may collect information from all or multiple of these lists and allocate tasks based on information from multiple lists.

[0103] The above-described systems and routines can reduce the cost of assigning tasks between processing nodes connected to a single server, and more specifically, between processing nodes of many corresponding servers connected to a server cluster. This is particularly advantageous for large server clusters, such as those included in data centers, where improvements in energy efficiency can have a large-scale impact on reducing costs and avoiding technical problems such as power outages. The reduced costs can be the result of reduced power consumed by selecting only the required processing devices and further by selecting groups of processing devices that are not yet occupied but whose many hardware and mechanical components are already in use and do not need to be activated or exit an idle or power-saving state. In addition, because the system and routines are configured to track and periodically update the available bandwidth of processing devices and groups of processing devices, and further because the system and routines are configured to track the availability of groups and assign tasks to available groups rather than queuing tasks with later available groups, the present disclosure also provides a method of reducing the time to complete assigned tasks.

[0104] Although the technology herein has been described with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present technology. Therefore, it should be understood that various modifications may be made to the exemplary embodiments, and other arrangements may be designed, without departing from the spirit and scope of the present technology as defined by the appended claims.

[0105] Most of the aforementioned alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, therefore, the aforementioned description of the embodiments should be carried out by way of example rather than by way of limiting the subject matter defined by the claims. As an example, it is not necessary to perform the aforementioned operations in the precise order described above. On the contrary, the various steps can be processed in different orders, such as inverted or simultaneously. Unless otherwise indicated, steps can also be omitted. In addition, the examples described herein and phrases such as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples, but rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method comprising: receiving, by one or more processors, a first task to be assigned among one or more of a plurality of processing devices; accessing, by the one or more processors, data indicating a plurality of predefined processing device groups, each processing device group including one or more processing devices, and the data further including bandwidth information indicating available bandwidth for each available processing device group and power information indicating an amount of power required to assign the first task to each processing device group, wherein the amount of power includes a combination of power required to operate the processing devices of the processing device groups and power required to operate cooling devices of the processing device groups; Assigning, by the one or more processors, the first task to a first processing device group having an available bandwidth greater than or equal to a required bandwidth for performing the first task based on power information associated with the first processing device group in the accessed data; as well as The power information of the accessed data is updated by the one or more processors in response to assigning the first task to the first group of processing devices.

2. The method according to claim 1, wherein: The first processing equipment group includes one or more servers, and wherein the cooling equipment includes at least one or more server fans for maintaining the temperature of the one or more servers.

3. The method according to claim 1, further comprising: receiving, by the one or more processors, a second task to be assigned among one or more of the plurality of processing devices; accessing, by the one or more processors, data indicative of the plurality of predefined groups of processing devices; assigning, by the one or more processors, the second task to a second processing device group having an available bandwidth greater than or equal to a required bandwidth for executing the second task based on power information associated with the second processing device group in the accessed data; as well as Power information of the accessed data is re-updated by the one or more processors in response to assigning the second task to the second group of processing devices. 4 . The method of claim 1 , further comprising updating the bandwidth information in response to assigning the first task to the first group of processing devices.

5. The method according to claim 4, further comprising: receiving, by the one or more processors, an indication that the first task has been completed; as well as The bandwidth information of the first processing device group is updated by the one or more processors to indicate that a bandwidth used by the first task is available.

6. The method according to claim 1, wherein: The one or more processing devices are connected to a common corresponding host server.

7. The method according to claim 6, wherein: The first task indicates each of a required host server bandwidth and a required peer bandwidth for performing the first task, wherein the data indicates each of available host server bandwidth and available peer bandwidth for each processing device group, and Wherein, assigning the first task to the first processing device group is further based on: The available host server bandwidth of the first processing device group is greater than or equal to the required host server bandwidth for performing the first task; and An available peer-to-peer bandwidth for the first group of processing devices is greater than or equal to a required peer-to-peer bandwidth for performing the first task.

8. The method according to claim 1, wherein: The data further indicates, for each processing device group, add power information indicating an amount of additional power required to allocate the first task to the processing device group, and The method further includes designating the available processing group having the lowest required amount of additional power as the first processing device group.

9. The method according to claim 8, wherein: Updating the data includes updating an amount of additional power required for one or more groups of processing devices that share at least one hardware component or mechanical component with the first group of processing devices.

10. The method according to claim 9, wherein: The power information and the added power information are determined based on a predetermined power requirement of at least one hardware component or mechanical component shared by two or more of the plurality of processing devices.

11. A system comprising: a data center comprising a plurality of processing devices, wherein at least some of the processing devices are configured to communicate with each other via a peer-to-peer connection; a memory for storing instructions and data, the data comprising a plurality of predefined subgroups of the plurality of processing devices, each subgroup comprising one or more processing devices, the data further comprising bandwidth information indicating available bandwidth for each of the subgroups, and power information indicating an amount of power required to perform tasks by each subgroup, wherein the amount of power comprises a combination of power required to operate the processing devices of the subgroup and power required to operate cooling devices of the subgroup; as well as one or more processors coupled to the memory and configured to execute stored instructions to: receiving a first task to be assigned to a subset of the plurality of processing devices; accessing, by the one or more processors, the bandwidth information and the power information associated with the plurality of subgroups; assigning, by the one or more processors, the first task to the first subgroup based on the bandwidth information and the power information associated with the first subgroup; and The power information associated with the first subset is updated, by the one or more processors, in response to assigning the first task to the first subset.

12. The system according to claim 11, wherein: The first subgroup includes one or more servers, and wherein the cooling device of the first subgroup includes at least one or more server fans for maintaining the temperature of the one or more servers.

13. The system according to claim 12, wherein: The one or more processors are further configured to execute the stored instructions to: receiving a second task to be assigned to a subset of the plurality of processing devices; accessing bandwidth information and power information associated with the plurality of subgroups; assigning the second task to a second subgroup having an available bandwidth greater than or equal to a bandwidth required for performing the second task based on the power information associated with the second subgroup; and The power information is re-updated in response to assigning the second task to the second subgroup.

14. The system according to claim 11, wherein: The one or more processors are configured to update the bandwidth information in response to assigning the first task to the first subgroup.

15. The system of claim 14, wherein: The one or more processors are further configured to execute the stored instructions to: receiving an indication that the first task has been completed; and The bandwidth information is updated to indicate that the bandwidth used by the first task is available.

16. The system of claim 15, wherein: The plurality of processing devices are connected to a common host server.

17. The system of claim 16, wherein: The first task indicates each of a required host server bandwidth and a required peer bandwidth for performing the first task, wherein the data indicates each of an available host server bandwidth and an available peer bandwidth for each available subgroup, and The one or more processors are configured to execute the stored instructions to assign the first task to the first subgroup further based on: The available host server bandwidth of the first processing device group is greater than or equal to the required host server bandwidth for performing the first task; and An available peer-to-peer bandwidth for the first group of processing devices is greater than or equal to a required peer-to-peer bandwidth for performing the first task.

18. The system of claim 11, wherein: The data further includes add power information indicating, for each subgroup, an amount of additional power required to allocate the first task to the subgroup, and The one or more processors are further configured to execute the stored instructions to designate the available processing group having the lowest required amount of additional power as the first subgroup.

19. The system of claim 18, wherein: The one or more processors are configured to execute the stored instructions to update the added power information of the first subgroup in response to assigning the first task to the first subgroup.

20. The system of claim 19, wherein: The one or more processors are configured to execute stored instructions to determine the power information and to add power information based on predetermined power requirements of at least one hardware component or mechanical component shared by two or more of the plurality of processing devices.

Images