switch

Abstract

A switch (10) relays requests that are transmitted from client computers (41-43) to one of a plurality of host computers (61-62). The switch (10) is provided with: an interface circuit (13) that accepts requests from the client computers (41-43); and a request processing circuit (19) that performs processing for determining which GPU of which host computer, among GPUs (70) provided in the plurality of host computers (61-62), is to execute each of the requests and for transmitting the request to the determined GPU. This configuration reduces the latency of requests.

Description

[Technical Field]

[0001] The present invention relates to a switch for use in a network. [Background technology]

[0002] There is known a technique for causing a request from a client computer to be executed by one of a plurality of processing circuits (for example, a GPU (Graphics Processing Unit)) of a host computer (Non-Patent Document 1). There is also known a technique for quickly processing each request by preparing a plurality of host computers and allocating the request to one of the processing circuits provided in the plurality of host computers. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ugljesa Milic et al. "S Beyond the Socket: NUMA-Aware GPUs," MICRO-50, October 14-18, 2017, Cambridge, MA, USA Summary of the Invention [Problem to be solved by the invention]

[0004] When multiple host computers are provided, it is conceivable to provide a schedule server separate from the multiple host computers that determines which processing circuit of which host computer should execute a request. However, when such a schedule server is provided, requests from clients must be sent to the schedule server once, which can result in an inconvenience of longer request latency.

[0005] An object of the present invention is to reduce the latency of requests. [Means for solving the problem]

[0006] In order to solve the above problem, the switch of the present invention is a switch that relays a request sent from a client computer to one of multiple host computers, and includes an interface circuit that accepts the request from the client computer, and a request processing circuit that determines which processing circuit of which host computer among the processing circuits provided in the multiple host computers should execute the request, and performs processing to send the request to the determined processing circuit. [Effects of the Invention]

[0007] The present invention reduces the latency of requests. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of a system in which a switch according to an embodiment of the present invention is used. [Figure 2] FIG. 2 is a diagram showing the configuration of a switch according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing the configuration of the latency table. [Figure 4] FIG. 4 is a diagram showing the configuration of the waiting time table. [Figure 5] FIG. 5 is a flowchart of the request processing. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] As shown in Fig. 1, a switch 10 according to the first embodiment of the present invention is connected to a plurality of client computers (hereinafter simply referred to as clients) 41 to 43. A router 50 connected to a network NW such as the Internet is also connected to the switch 10. A plurality of host computers (hereinafter simply referred to as hosts) 61 and 62 are connected to the network NW via a router (not shown) or the like. The number of clients and hosts is arbitrary.

[0011] The host 61 includes a CPU (Central Processing Unit) 61A, a RAM (Random Access Memory) 61B that functions as the main memory of the CPU 61A, a non-volatile memory 61C, a NIC 61D, GPUs (Graphics Processing Units) 71 to 73, and a PCIe bus 61G to which these are connected. Each of the GPUs 71 to 73 is formed by an FPGA (Field-Programmable Gate Array). The CPU 61A performs the following operations by executing a program stored in the non-volatile memory 61C. For example, the CPU 61A can individually compile the GPUs 71 to 73. Through this compilation, at least one of models A to D for request execution, which will be described later, is configured in each of the GPUs 71 to 73.

[0012] The host 62 includes a CPU 62A, a RAM 62B that functions as the main memory of the CPU 62A, a non-volatile memory 62C, a NIC 62D, GPUs 74 to 76, and a PCIe bus 62G to which these are connected. Each of the GPUs 74 to 76 is configured by an FPGA. The operation of the host 62 is similar to that of the host 61, and therefore a detailed description thereof will be omitted.

[0013] In the following description, the clients 41 to 43 are also collectively referred to as clients 40. The hosts 61 and 60 are also collectively referred to as hosts 60. The GPUs 71 to 76 are also collectively referred to as GPUs 70.

[0014] The client 40 transmits, as its own transmission data, a request to cause the GPU 70 of the host 60 to execute processing, or a non-request communication packet exchanged in communication with another client 40 or another computer connected via the network NW. Whether the transmission data transmitted by the client 40 is a request or a communication packet can be identified, for example, by the header of the transmission data. For example, request specifying information that specifies that the transmission data is a request is stored in the header of the transmission data. Transmission data that does not include this request specifying information is a communication packet.

[0015] As shown in FIG. 2, the switch 10 includes a plurality of ports P1 to Pn (n is the total number of ports), a controller 11, a communication packet buffer 12A, a request buffer 12B, an interface circuit 13, a table memory 14, a communication packet processing circuit 18, and a request processing circuit 19. The numerical portion of the symbols of ports P1 to Pn is the port number of each port. For example, the port number of port P3 is "3." Ports P1 to Pn are also collectively referred to as ports P.

[0016] The ports P1 to P3 are connected to clients 41 to 43, respectively. The port P4 is connected to a router 50.

[0017] The controller 11 includes a CPU (Central Processing Unit) 11A, a RAM (Random Access Memory) 11B that is the main memory of the CPU 11A, and a non-volatile memory 11C that stores programs and data that cause the CPU 11A to operate as described below. The controller 11 controls the entire switch 10 and also performs the operations described below.

[0018] Communication packets are stored in the communication packet buffer 12 A. Requests are stored in the request buffer 12 B. Each of the buffers 12 A and 12 B is configured by a RAM.

[0019] Transmission data from the client 40 or the host 60 is input to the interface circuit 13 via port P. The interface circuit 13 checks the header of the input transmission data and determines whether the transmission data is a request or a communication packet.

[0020] If the input transmission data is a communication packet, the interface circuit 13 stores the communication packet in the communication packet buffer 12 A. If the input transmission data is a request, the interface circuit 13 stores the request in the request buffer 12 B. The communication packet buffer 12 A and the request buffer 12 B are configured in a queue structure.

[0021] The table memory 14 is configured by RAM and stores a MAC (Media Access Control address) table, a latency table, a waiting time table, etc. These tables will be described later.

[0022] The communication packet processing circuit 18 sequentially acquires (dequeues) communication packets from the communication packet buffer 12A. The communication packet processing circuit 18 refers to the MAC table in the table memory 14. The MAC table indicates the correspondence between MAC addresses and port numbers. The communication packet processing circuit 18 acquires from the MAC table the port number corresponding to the destination MAC address included in the header of the communication packet, and transfers the communication packet to port P of the acquired port number. As a result, the communication packet is transmitted to the device having the MAC address. Such a transfer operation is a normal operation of the switch 10.

[0023] The request processing circuit 19 sequentially acquires (dequeues) requests from the request buffer 12B. The request processing circuit 19 transmits the request to the GPU 70 that can optimally process the request (with the shortest waiting time) at the time of acquisition. Details of the request processing circuit 19 will be described later.

[0024] The buffer memory 12, the interface circuit 13, the table memory 14, the communication packet processing circuit 18, and the request processing circuit 19 are configured, for example, in an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). In this case, the buffer memory 12 and the table memory 14 are configured in a BRAM (Block Random Access Memory) or an SRAM (Static Random Access Memory) provided in the ASIC or FPGA.

[0025] The request processing circuit 19 refers to the latency table and waiting time table in the table memory 14 to identify the destination of the request acquired from the request buffer 12B, that is, the GPU 70 of the host 60 that will execute the request.

[0026] A request is a command to cause the GPU 70 to execute processing on data to be processed. The request includes a request ID for identifying itself, the MAC address and IP address of the client 40 that sent the request, the data to be processed, a model ID for identifying a model that processes the data, and a latency limit time required by the client 40 from the transmission of the request to the reception of the execution result of the request. The request ID may be assigned by the switch 10. By executing the request, the GPU 70 processes the data to be processed using the model identified by the model ID.

[0027] As shown in FIG. 3, the latency table indicates, for each record, a model currently configured in the GPU 70 and its latency. In the latency table, a model ID, a GPU ID identifying the GPU 70 in which the model identified by this model ID is configured, a host ID identifying the host 60 having this GPU 70, and the latency of the model when processing is executed are associated with each other. In FIG. 3, the model ID is represented by the alphabet (any of "A" to "D") of models A to D, and the GPU ID is represented by a code (any of "70" to "76") assigned to the GPU 70. The host ID is represented by a code ("61" or "62") assigned to the host 60 (similar to FIG. 4). In the example of FIG. 3, models A and D are configured in the GPU 71, and the respective latencies are a1 [msec] and a2 [msec].

[0028] When a model is newly configured in the GPU 70, one record indicating the model is added to the latency table. On the other hand, when a model is deleted from the GPU 70, one record indicating the model is deleted. Addition and deletion of one record are performed by the CPU 11A.

[0029] As shown in FIG. 4, each record in the latency table identifies a request that has been sent to the host 60 and for which the execution result (processing result) of the request has not yet been returned. Each record in the latency table corresponds to a host ID identifying the host 60 to which the request was sent, a model ID identifying the model executing the request, a GPU ID identifying the GPU 70 executing the request, a latency indicating the processing time required to execute the request, and a request ID for the request. The host ID, GPU ID, and client ID may be represented by an IP address or MAC address. Note that the GPU 70 cannot process each model in parallel. In the example of FIG. 4, two requests are currently being processed or are waiting to be processed, with the GPU 75 of the host 62 executing model D and model C (request IDs "0030" and "0032"). The two requests will be completed after a maximum latency of b1 + b3 has elapsed.

[0030] One record in the waiting time table is added when a processing request is sent to the GPU 70, and is deleted when the execution result of the request is returned. Addition and deletion of one record are performed by the CPU 11A.

[0031] The request processing circuit 19 executes, for example, the request processing shown in FIG. 5. First, the request processing circuit 19 refers to the latency table and acquires all GPU IDs and latencies corresponding to the model IDs included in the request for each model ID (step S11). The request processing circuit 19 also refers to the wait time table and acquires all wait times corresponding to the acquired GPU IDs for each GPU ID (step S12). The request processing circuit 19 sums up the wait times and latencies for each GPU 70 to acquire a total wait time (step S13). This total wait time is the predicted time until the request processing is completed, that is, until the execution result of the request is returned to the client 40.

[0032] Thereafter, the request processing circuit 19 identifies a GPU 70 for which the total waiting time calculated in step S13 is shorter than the time limit included in the request (step S14). The time limit is the time from when the client 40 transmits a request until the execution result is returned to the client 40 (request latency). Note that processing time other than the above-mentioned waiting time, for example, the time required to process a request in the switch 10, is shorter than the above-mentioned waiting time and does not need to be taken into consideration in the above comparison, but the time obtained by adding this predicted processing time to the total waiting time may be compared with the time limit.

[0033] The request processing circuit 19 performs a process of transmitting a request to an arbitrary GPU 70 among the one or more GPUs 70 identified above (step S15). The arbitrary GPU 70 may be, for example, the GPU 70 with the shortest total latency, or a randomly selected GPU. When transmitting the request, the GPU ID (which may be an IP address, etc.) of the destination GPU 70 and the MAC address and IP address of the host 60 equipped with the GPU 70 are also transmitted together with the request. In step S15, the request processing circuit 19 transfers the transmission data, such as the request, to port P4, to which the router 50 is connected, via the interface circuit 13. The CPU 11A communicates with the request processing circuit 19 to obtain necessary data, creates one record in a latency table based on the obtained data, and adds the created record to the latency table. The "latency" included in one record is the latency obtained above.

[0034] The requests transferred to port P4 are sent to the host 60 having the destination IP address via the router 50 or the like. The requests are stored in a queue-structured request buffer provided for each GPU 70 of the host 60. The CPU 61A or the like of the host 60 sequentially acquires the requests from the request buffer and, based on the acquired requests, causes the GPU 70 identified by the GPU ID to execute the requests. Specifically, the CPU 61A or the like causes the GPU 70 identified by the GPU ID to execute processing on the data to be processed in the model identified by the model ID. In this way, the requests are sequentially executed for each GPU 70. The host 60 returns the execution results of the requests by the GPU 70 to the switch 10, along with the request ID included in the request and the MAC address and IP address of the client 40 that sent the request. Note that the requests and their execution results may be exchanged directly between the NIC 61D and the GPU 70 without going through the CPU 61A or the like.

[0035] The interface circuit 13 of the switch 10 transfers the execution results and the like to the request processing circuit 19. The request processing circuit 19 transfers the request ID included in the execution results and the like to the CPU 11A (step S16). The CPU 11A deletes one record including the transferred request ID from the waiting time table. Furthermore, the request processing circuit 19 generates a packet in which information such as the MAC address and IP address of the sending client 40 is added as a header to the execution results (step S17). The request processing circuit 19 refers to the MAC table and transfers the packet via the interface circuit 13 to port P, which has a port number corresponding to the MAC address of the sending client 40 of the request (step S18). As a result, the execution results are returned to the sending client 40 of the request.

[0036] In step S14, there may be no GPU 70 whose total waiting time calculated in step S13 is shorter than the time limit included in the request. In this case, the CPU 11A is notified of this fact. The CPU 11A is assumed to be able to communicate with the host 60 and to be able to grasp the free space of memory, etc. of each GPU 70. In this case, the CPU 11A transmits to the host 60 an instruction to deploy the requested model to a GPU 70 whose free space is relatively large and whose total waiting time in the waiting time table is shorter than a predetermined threshold. If the deployment is successful, the host 60 notifies the CPU 11A of this fact. In this case, the host 60 notifies the CPU 11A of the latency predicted during deployment. The CPU 11A stores the notified latency together with the model ID, GPU ID, etc. as one record in the latency table. Thereafter, the processing from step S12 onwards may be performed again. Note that the total waiting time compared with the time limit at this time may include the time required to deploy the model, as appropriate.

[0037] As described above, the switch 10 according to this embodiment relays a request transmitted from a client computer 40 to one of the plurality of host computers 60. The interface circuit 13 accepts a request from the client computer 40. The request processing circuit 19 determines which GPU 70 of which host 60, among the GPUs 70 provided in the plurality of host computers 60, should execute the request, and transmits the request to the determined GPU 70. With this configuration, the switch 10 completes the process of identifying the destination of the request, eliminating the need to transmit the request to another server. This reduces the latency of the request. Furthermore, the utilization efficiency of GPUs 70 across the plurality of hosts 60 is improved.

[0038] Furthermore, the request processing circuit 19 acquires a predicted time (total waiting time) until the execution of the request is completed in each GPU 70, and performs processing to transmit the request to the GPU 70 for which the acquired predicted time is shorter than a predetermined standard (time limit). This shortens the latency of the request. Note that the predetermined standard may be a predetermined fixed time, etc.

[0039] The switch 10 includes a table memory 14 that stores a latency table for managing each latency until the processing of each GPU is completed, and a CPU 11A that updates each latency in the latency table. The request processing circuit 19 acquires at least a portion of each latency in the latency table (the latency of the GPU for which a request model is configured) and performs processing to acquire a predicted time based on at least a portion of the acquired latency, and is configured with an ASIC or FPGA. In the switch 10, a matching circuit that references a MAC table or an IPv4 table (not shown) is configured with an ASIC or FPGA, and table matching can be performed at high speed. With the above configuration, the request processing circuit 19 is configured with an ASIC or FPGA, and therefore the processing by the request processing circuit 19 (matching processing using the latency table) is accelerated by the matching circuit in the same way as table matching.

[0040] The switch 10 also includes an instruction unit (in the above example, this is the CPU 11A, but it may also be the request processing circuit 19) that instructs the host computer 60 having the FPGA to configure one of the models A to D in the FPGA, which is the GPU 70. This allows the model to be dynamically rewritten.

[0041] Furthermore, when there is no processing circuit whose acquired prediction time is shorter than the predetermined standard, the instruction unit deploys the model to the host computer 60. This makes it possible to send a request to the GPU 70 with the shortest prediction time.

[0042] The above embodiment can be modified in various ways. For example, the circuit that executes the host processing may be a processing circuit (particularly an accelerator) such as a processor other than a GPU. The content of the request is also arbitrary. Port P may include a port to which an in-house LAN or the like is connected. The switch may be a switching hub, a network switch, or the like.

[0043] The present invention is not limited to the above-described embodiments and modifications. For example, the present invention includes various modifications to the above-described embodiments and modifications that are understandable to those skilled in the art within the scope of the technical concept of the present invention. The configurations listed in the above-described embodiments and modifications can be combined as appropriate to the extent that no contradictions exist. Furthermore, any of the above-described configurations can be deleted. The various programs described above may be stored in a non-transitory computer-readable storage medium, not limited to the non-volatile memory 61C. A "device" and a "unit" may be an entity in which the configuration that realizes the operation is housed in a single housing, or an entity (system) in which the configuration that realizes the operation is housed distributed across multiple housings.

[0044] (Addendum) The configurations disclosed in this specification, taking the above-described embodiments and modifications as examples, will be exemplified below. (Appendix 1) A switch that relays a request sent from a client computer to one of multiple host computers, comprising: an interface circuit that accepts the request from the client computer; and a request processing circuit that determines which processing circuit of which host computer among the processing circuits provided in the multiple host computers should execute the request, and performs processing to send the request to the determined processing circuit. (Appendix 2) The request processing circuit obtains a predicted time until the execution of the request in each of the processing circuits is completed, and performs the process of sending the request to one of the processing circuits for which the obtained predicted time is shorter than a predetermined standard. (Appendix 3) 3. The switch of claim 1, wherein the request includes a time limit for returning the execution result of the request to the client computer, and the request processing circuit sends the request to the processing circuit whose predicted time is within the time limit. (Appendix 4) A switch according to any one of appendices 1 to 3, comprising: a memory having a table for managing each waiting time until the processing of each of the processing circuits is completed; and a CPU (Central Processing Unit) for updating each of the waiting times in the table, wherein the request processing circuit acquires at least a portion of each waiting time in the table and performs processing to acquire the predicted time based on at least a portion of the acquired waiting times, and is configured using an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). (Appendix 5) A switch according to any one of appendices 1 to 4, wherein at least one of the processing circuits is an FPGA and is capable of configuring a plurality of models, and the switch includes an instruction unit that instructs a host computer having the FPGA, among the plurality of host computers, to configure one of the plurality of models in the FPGA. (Appendix 6) A switch as described in any one of appendices 1 to 5, wherein the request processing circuit acquires a predicted time until the execution of the request is completed in each of the processing circuits, and performs the process of sending the request to a processing circuit among the processing circuits for which the acquired predicted time is shorter than a predetermined standard, and the instruction unit issues the instruction to the host computer when there is no processing circuit for which the acquired predicted time is shorter than the predetermined standard. [Explanation of symbols]

[0045] 10...switch, 11A...CPU, 12A...communication packet buffer, 12B...request buffer, 13...interface circuit, 14...table memory, 18...communication packet processing circuit, 19...request processing circuit, 40-43...client computer, 50...router, 60-62...host computer, 70-76...GPU.

Claims

1. A switch that relays a request transmitted from a client computer to one of a plurality of host computers, an interface circuit for receiving the request from a client computer; a request processing circuit that determines which processing circuit of which host computer should execute the request among the processing circuits provided in a plurality of host computers, and transmits the request to the determined processing circuit; a memory provided with a table for managing each waiting time until the processing of each of the processing circuits is completed; a CPU (Central Processing Unit) that updates each of the waiting times in the table; The request processing circuit As the processing, at least a part of each waiting time in the table is acquired, and a predicted time until the execution of the request in each of the processing circuits is completed is acquired based on at least a part of the acquired waiting time, and the request is transmitted to a processing circuit among the processing circuits for which the acquired predicted time is shorter than a predetermined standard; and It is configured with an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), switch.

2. the request includes a time limit for returning the execution result of the request to the client computer; the request processing circuit transmits the request to the processing circuit for which the predicted time is within the time limit. The switch of claim 1 .

3. At least one of the processing circuits is an FPGA, and a plurality of models can be configured; the switch includes an instruction unit that issues an instruction to a host computer having the FPGA, among the plurality of host computers, to configure one of the plurality of models in the FPGA; The switch of claim 1 .

4. The instruction unit issues the instruction to the host computer when there is no processing circuit for which the acquired predicted time is shorter than the predetermined standard. The switch of claim 3.

Images