Computer device

Abstract

A computer device (10) comprises: a GPU (50); and a NIC (60) that, through direct memory access (DMA), transfers a plurality of pieces of data to the GPU (50). The GPU (50) is provided with a plurality of kernels 151 that respectively execute processes in parallel on a plurality of pieces of data DMA-transferred from the NIC (60). The NIC (60) controls the GPU (60) so as to increase the number of kernels 151 when a prescribed condition that the number of kernels 151 is insufficient with respect to the number of the plurality of pieces of data is satisfied. The GPU (60) increases the number of kernels 151 by starting a program prepared in advance.

Description

computer equipment

[0001] The present invention relates to computer devices.

[0002] A computer device is known that includes a first device and a second device that performs DMA data transfer to the first device, and the first device includes multiple arithmetic units that each execute processing in parallel on multiple pieces of data that have been DMA transferred from the second device (Non-Patent Document 1).

[0003] NVIDIA, "DOCA GPUNetIO", [online], Last updated on Feb 9, 2024, [Retrieved August 6, 2024], Internet <URL: https: / / docs.nvidia.com / doca / sdk / doca+gpunetio / index.html>

[0004] In the above computer device, even if the amount of data to be DMA transferred increases, the number of processing units cannot be changed, which causes the processing by the processing units to be unable to keep up, resulting in an increased latency.

[0005] The present invention aims to reduce the increase in latency.

[0006] The computer device of the present invention comprises a first device and a second device that transfers multiple pieces of data to the first device via DMA (Direct Memory Access), the first device having multiple arithmetic units that each execute processing in parallel on the multiple pieces of data DMA transferred from the second device, and the second device controls the first device to increase the number of the multiple arithmetic units when a predetermined condition is met in which the number of the multiple arithmetic units is insufficient for the number of the multiple pieces of data.

[0007] According to the above configuration, an increase in latency is reduced.

[0008] Fig. 1 is a diagram showing the configuration of a computer device according to an embodiment of the present invention. Fig. 2 is a diagram showing the configuration of a NIC and a GPU according to an embodiment. Fig. 3 is a diagram showing the configuration of a NIC and a GPU according to an embodiment. Fig. 4 is a diagram showing the configuration of a table developed in the RAM of the NIC.

[0009] Hereinafter, a computer device according to an embodiment of the present invention will be described with reference to the drawings. Note that in each drawing, only some of the elements may be labeled with a reference numeral. Furthermore, multiple elements that correspond to each other and have the same or similar functions may be labeled with the same reference numeral.

[0010] The computer device 10 according to this embodiment shown in FIG. 1 is a server computer that performs inference processing based on data supplied from multiple client computers CC via a network NW such as the Internet or a local area network. The computer device 10 returns the results of the inference processing to the client computers CC. An example of the data is image data, and an example of the inference processing is labeling of subjects depicted in the image data. The computer device 10 has two types of inference models A and B. The inference models A and B are, for example, deep neural networks (DNNs), and execute different inference processes A and B, respectively. The client computers CC specify which inference model to use. The number of inference models and the content of the inference processing are arbitrary.

[0011] The computer device 10 includes a processor 20, a storage 30, a random access memory (RAM) 40, a graphics processing unit (GPU) 50, a network interface card (NIC) 60, and a bus 70. The inference processes A and B described above are assumed to be executed by the GPU 50. The elements 20 to 60 are connected to a PCIe standard bus 70 and are capable of communicating with each other. In particular, the GPU 50 and the NIC 60 can communicate directly via the bus 70 without going through the elements 20 to 40.

[0012] The processor 20 includes a CPU (Central Processing Unit) and other components. The processor 20 executes or uses programs and data stored in the non-volatile storage 30 to control the entire computer device 10 and perform predetermined processes. The RAM 40 functions as the main memory of the processor 20. The processor 20 can communicate with external devices such as client computers via the network NW and the NIC 60 connected thereto to perform predetermined processes.

[0013] Various settings are made to the GPU 50 using CUDA. The GPU 50 includes a memory 52. ​​The memory 52 may be located outside the GPU 50. In this case, the memory 52 is configured as a GPU memory that communicates with the GPU 50 via a bus 70.

[0014] The memory 52 stores a plurality of programs (more specifically, binary files) that are compiled and executed by the GPU 50. The programs may be stored in the storage 30 and read into the memory 52 as needed. The GPU 50 can operate as each of a plurality of types of arithmetic units depending on the type of program being executed. The plurality of arithmetic units include a receiving kernel 151, a first processing execution kernel 152, a second processing execution kernel 153, and a sending kernel 154, as shown in FIG. 2 .

[0015] 1 and 2, the NIC 60 includes a NIC core 61 that processes packets input to the NIC 60, and a DMA engine 62 that performs DMA (Direct Memory Access) transfer between the NIC 60 and the GPU 50. The NIC 60 also includes a CPU 63 and a RAM 64, which will be described later.

[0016] The basic flow of inference processing performed by the GPU 50 using the above configuration will now be described with reference to Figure 2. It is assumed that multiple pieces of data to be subjected to inference processing are divided into packets and input to the computer device 10 from multiple client computers CC. The GPU 50 includes two receiving kernels 151, two first processing execution kernels 152, two second processing execution kernels 153, and two sending kernels 154. Each kernel is realized by the GPU 50 executing a program that realizes that kernel. Each kernel sequentially processes the following data:

[0017] The NIC core 61 of the NIC 60 receives a packet transmitted from the client computer CC via the network NW and analyzes the header of the received packet. Based on the analysis, the NIC core 61 transfers the packet to the destination indicated by the information stored in the header. When the GPU 50 is to perform inference processing, the packet header contains a model ID that identifies the inference model for which inference is to be performed. If the model ID is included in the header, the NIC core 61 issues an inference request ID. The NIC core 61 writes the same request ID in the header of each packet into which the same data is divided. The request ID may be issued by the client computer CC and written in the header. The NIC core 61 instructs the DMA engine 62 to transfer the packet, whose header contains the model ID and request ID, to the receive buffer 52A provided in the memory 52 of the GPU 50. The DMA engine 62 communicates with the GPU 50 and DMA-transfers the packet to the receive buffer 52A. The above-described processing for packets is performed sequentially for each packet that is sequentially input to the NIC 60 .

[0018] Packets from the NIC 60 are stored in the receive buffer 52A, for example, in chronological order. The two receive kernels 151 operate in parallel and sequentially read packets from the receive buffer 52A. At this time, the receive kernel 151 reads packets containing the same request ID, starting with the oldest. The receive kernel 151 extracts information from the packet header and generates metadata (see below for details) based on the extracted information. The receive kernel 151 adds the generated metadata to the packet payload. The receive kernel 151 stores the payload data with the metadata added as first data in the workspace 52B provided in the memory 52. ​​The receive kernel 151 reserves a predetermined storage space in the workspace 52B and stores multiple pieces of first data with the same request ID in the reserved storage space in chronological order. As a result, multiple pieces of first data with the same request ID are stored in each storage space. By concatenating the payload portions of these multiple pieces of first data in chronological order, a block of data to be subjected to inference processing is obtained. The metadata includes the following information: Source information (such as the address of the source of the packet) Destination information (such as the address of the destination of the packet) Model ID Area specification information (for example, the starting address and range) that specifies the storage area in workspace 52B that was secured when the first data was stored

[0019] The receiving kernel 151 transmits the generated metadata to the first process execution kernel 152 or the second process execution kernel 153. At this time, if the model ID included in the metadata specifies inference model A, the receiving kernel 151 supplies the metadata to one of the two first process execution kernels 152 that is in a standby state (for example, a state in which the previous process has ended). If the model ID included in the metadata specifies inference model B, the receiving kernel 151 supplies the metadata to one of the two second process execution kernels 153 that is in a standby state.

[0020] The process execution kernel 152 or 153, to which the metadata has been supplied, sequentially retrieves the first data in chronological order from the storage area identified by the area identification information included in the metadata. The process execution kernel 152 or 153 connects together the payload portions included in the sequentially retrieved first data in chronological order to obtain the target data for the inference process. The process execution kernel 152 or 153 performs inference processing on the obtained target data using an inference model, and stores the inference results together with the metadata in the workspace 52B as second data. The metadata includes the following information: Source information, destination information, model ID, and area identification information (e.g., starting address and range) that identifies the storage area for the second data in the workspace 52B.

[0021] The two first process execution kernels 152 can execute inference process A separately and in parallel on different target data. The two second process execution kernels 153 can execute inference process B separately and in parallel on different target data.

[0022] The process execution kernel 152 or 153 transmits the generated metadata to one of the two transmission kernels 154 that is in a standby state. The transmission kernel 154 acquires the second data from the storage area specified by the area specification information of the received metadata, converts (divides) the second data into one or more packets, and stores the converted packets in a transmission buffer 52C provided in the memory 52. ​​The payload of the packet stores all or part of the inference result after division. The header of the packet stores information in which the source information and destination information of the metadata are converted into destination information and source information. The transmission kernels 154 can operate in parallel.

[0023] A plurality of packets are stored in the transmission buffer 52C. These packets are transferred to the NIC core 61 by the DMA engine 62 of the NIC 60, and are then returned from the NIC core 61 to the destination client computer CC indicated by the destination information in the header. As a result, the inference result inferred by the computer device 10 is returned to the client computer CC.

[0024] With the above configuration, the GPU 50 can perform various processes in parallel, but if the amount of data to be processed received by the NIC 60 increases, the number of kernels for parallel processing may become insufficient, resulting in problems such as increased latency in the GPU 50 or the computer device 10. Therefore, in this embodiment, the number of kernels is increased as needed, thereby reducing the increase in latency (including preventing the increase in latency).

[0025] To increase the number of kernels, a required number of compiled programs for each kernel are prepared in the memory 52 (or storage 30) of the GPU 50. The number of programs may be determined appropriately by a user of the GPU 50 (for example, a service provider that provides inference processing services or an administrator that manages the computer device 10) depending on the content of the inference processing, the connection environment of the computer device 10, etc.

[0026] After the above programs are prepared, the GPU 50 launches the prepared programs (i.e., starts executing the programs) in response to a predetermined first trigger. This increases the number of kernels by the number of launched programs. If the number of kernels is excessive, a second trigger may be triggered to preempt (forcefully terminate) launched kernels (e.g., launched programs). This reduces the number of kernels. The reduction of kernels may be achieved, for example, by terminating program execution and is not limited to the aforementioned forced termination. Here, the kernels may be CUDA kernels, and the preemption itself may be achieved by a known method. Furthermore, the method of launching a program (CUDA binary) itself may employ a known method (e.g., CUDA Graph) in which an already launched CUDA kernel launches another CUDA kernel.

[0027] The method for increasing or decreasing the number of kernels will be described in detail with reference to Fig. 3. The RAM 64 of the NIC 60 in Fig. 3 stores a table 64A. The table 64A is stored in a non-volatile memory (not shown) of the NIC 60 or in the storage 30, and is loaded into the RAM 64 at an appropriate timing.

[0028] As shown in FIG. 4 , table 64A shows the relationship between the model ID of an inference model (here, "A" or "B" is assigned to the inference process) and the number of kernels recommended for executing the inference process using that inference model. The number of kernels (the number of parallel processes) required to prevent processing delays in the GPU 50 when performing inference processing for an inference model can be predicted to some extent experimentally or logically. This is particularly true when the inference process is a DNN. This is because a DNN has few branches and the input data rate, calculation intensity (Floating Point per second), and transmission data rate are basically determined at compile time. Because such predictions are possible, a service provider providing an inference processing service or an administrator managing the computer device 10 is expected to derive the number of kernels required for each inference model and create table 64A.

[0029] 3 , the CPU 63 of the NIC 60 monitors packets passing through the NIC core 61 and counts the number of inference processes currently being executed by the GPU 50 (i.e., the number of process execution kernels 152 and 153 in operation) for each type of inference process. For example, when a model ID is included in the header of a packet input to the NIC core 61 (e.g., the first packet of a series of packets into which transmission data is divided), the CPU 63 increments by one the number of inference processes being executed for the inference model identified by the model ID. Also, when a model ID is included in the header of a packet output from the NIC core 61 to the outside, the CPU 63 decrements by one the number of inference processes being executed for the inference model identified by the model ID.

[0030] The CPU 63 references the table 64A at any timing to obtain the number of kernels corresponding to model IDs with an execution count of 1 or more. Then, for each model ID with an execution count of 1 or more, the CPU 63 calculates a value by multiplying the number of kernels by the execution count and adds up the calculated values ​​for each kernel. This sum is the number of preferred kernels. For example, the counted number of executions of the inference process for the model ID "A" indicating inference process A is set to "2," and the counted number of executions of the inference process for the model ID "B" indicating inference process B is set to "3." In this case, the preferred number of reception kernels 151 is 2 x 4 + 3 x 3 = 17. Such a number (i.e., the required number of kernels) is calculated for each kernel and provided to the NIC core 61.

[0031] The NIC core 61 writes the required number of each kernel supplied from the CPU 63 into the header of a packet having a model ID in the header. This packet is DMA-transferred to the GPU 50 as described above. When generating metadata based on the packet header, the receiving kernel 151 includes the required number of each kernel in the metadata. This metadata propagates to each kernel. Each kernel compares the required number of kernels included in the metadata with the current number of kernels and activates the missing number of kernels (the kernels indicated by dotted lines in FIG. 3 are examples of added kernels). Specifically, the GPU 50 activates (in other words, starts execution of) the missing number of programs among the programs that realize that kernel. Note that a kernel such as the receiving kernel 151 may activate other (subsequent) kernels (e.g., the first processing execution kernel 152 for the receiving kernel 151, the sending kernel 154 for the first processing execution kernel 152, etc.) by the difference between the required number and the actual number of the subsequent kernels. If the actual number is greater than the required number, the kernel may be terminated. The NIC core 61 may also identify a required kernel from the model ID of an incoming packet, and if a kernel that is unnecessary for a certain period of time exists, preempt the kernel.

[0032] According to this embodiment, whether or not to increase the number of kernels is determined based on an initially prepared table, so the number of kernels can be quickly increased or decreased in response to changes in the situation. Furthermore, the amount of calculation required for the determination is small, so the burden on the NIC 60 is small. Increasing the number of kernels increases the throughput of the computer device 10 and reduces increases in latency. Furthermore, reducing the number of kernels prevents unnecessary kernels from being launched, reducing power consumption.

[0033] In a first modification, the NIC 60 has a function of counting the number of passing packets per unit time. Therefore, for example, the NIC core 61 may count the number of passing packets having a model ID in their header per unit time, and increase or decrease the number of receiving kernels 151 according to the count. When the count increases, the NIC core 61 controls the GPU 60 to increase the number of receiving kernels 151. Furthermore, when the count decreases, the NIC core 61 controls the GPU 60 to decrease the number of receiving kernels 151. This control is performed by writing the number of receiving kernels 151 in the header of the packet, as described above. As another example of this control, the increment or decrement of the number may be written in the header. When decreasing the receiving kernels 151, preemption may be used.

[0034] As a second modification, a typical GPU 60 constantly measures its own operating status (resource usage status, processing efficiency, etc.). Therefore, the CPU 63 of the NIC 60 may communicate with the GPU 60 to acquire the operating status of the GPU 60. The CPU 63 then controls the GPU 60 to increase the number of process execution kernels 152 or 153 (writing an indication to increase the number in the header) while monitoring the acquired operating status. The CPU 63 may then control the increase of the process execution kernels 152 or 153 until the increase in the process execution kernels 152 or 153 does not change the operating status of the GPU 60.

[0035] As a third variant, the NIC core 61 may monitor the transmission rate of packets from the GPU 60 and control the number of transmitting kernels 154 so that the transmission rate is maximized (such as by writing an increase or decrease in the number of transmitting kernels 154 to the header).

[0036] In each of the above processes, preemption may be used to reduce the kernel.

[0037] By performing at least one of the processes in the other examples described above, the number of kernels can be set to an optimum value in accordance with the current situation, and effects such as increased throughput, reduced processing latency, and reduced power consumption can be obtained.

[0038] The present invention is not limited to the above-described embodiments and modifications. For example, the present invention includes various modifications to the above-described embodiments and modifications that can be understood by a person skilled in the art within the scope of the technical concept of the present invention. The configurations listed in the above-described embodiments and modifications can be combined as appropriate within a range that does not cause contradictions. In addition, any of the above-described configurations can be deleted.

[0039] (Notes) The following are examples of the above-described embodiments and modifications. For each note, only a partial configuration of the above-described embodiments and modifications may be applied. Furthermore, parts of each note may be combined.

[0040] (Supplementary Note 1) A computer apparatus comprising: a first device (e.g., GPU 60); and a second device (e.g., NIC 50) that transfers a plurality of data items to the first device via DMA (Direct Memory Access), wherein the first device comprises a plurality of arithmetic units (e.g., any of kernels 151 to 154) that each execute a process in parallel on the plurality of data items DMA-transferred from the second device (e.g., the plurality of data items may be data items that have actually been transferred, or data items that have been processed), and wherein the second device controls the first device to increase the number of the plurality of arithmetic units (e.g., by writing the number of kernels to a header or writing an increase / decrease in the number) when a predetermined condition is met in which the number of the plurality of arithmetic units is insufficient for the number of the plurality of data items (e.g., when the required number of kernels calculated based on table 64A is greater than the current number of kernels, or when the number of kernels is increased in any of variants 1 to 3). (Supplementary Note 2) The computer apparatus according to Supplementary Note 1, wherein the first device is a GPU (Graphics Processing Unit), and each of the plurality of arithmetic units includes the GPU that executes a program. (Supplementary Note 3) The computer device according to Supplementary Note 2, wherein the GPU increases the number of the plurality of arithmetic units by starting a pre-prepared program (for example, the GPU starts a program to operate as a new arithmetic unit, thereby increasing the number of arithmetic units). (Supplementary Note 4) The computer device according to Supplementary Note 2 or 3, wherein the GPU decreases the number of the plurality of arithmetic units by terminating execution of the program. (Supplementary Note 5) The computer device according to any of Supplements 1 to 4, wherein the second device is a NIC. (Supplementary Note 6) The computer device according to any of Supplements 1 to 5, wherein the second device obtains the type of the new process to be performed, refers to a table (for example, table 64A) indicating the relationship between the type of process and the number of arithmetic units required for this type based on the obtained type, obtains the number of arithmetic units corresponding to the obtained type, and increases the number of the plurality of arithmetic units based on the obtained number.(Supplementary Note 7) The computer device according to any one of Supplementary Notes 1 to 6, wherein the second device monitors a data processing status of the first device (the number of packets passing through the NIC 60 per unit time, the operating status of the GPU 60, or the packet transmission rate), and increases the number of the plurality of arithmetic units when the status satisfies the predetermined condition.

[0041] 10...Computer device, 20...Processor, 30...Storage, 40...RAM, 50...GPU, 52...Memory, 52A...Receive buffer, 52B...Workspace, 52C...Transmit buffer, 60...NIC, 61...NIC core, 62...DMA engine, 63...RAM, 64A...Table, 70...Bus, 151...Receiving kernel, 152...First processing execution kernel, 153...Second processing execution kernel, 154...Transmitting kernel, CC...Client computer, NW...Network.

Claims

1. A computer device comprising: a first device; and a second device that transfers multiple pieces of data to the first device via DMA (Direct Memory Access), wherein the first device comprises multiple arithmetic units that each execute processing in parallel on the multiple pieces of data DMA-transferred from the second device, and the second device controls the first device to increase the number of the multiple arithmetic units when a predetermined condition is met that causes the number of the multiple arithmetic units to be insufficient compared to the number of the multiple pieces of data.

2. The computer device of claim 1, wherein the first device is a GPU (Graphics Processing Unit), each of the plurality of arithmetic units includes the GPU that executes a program, and the GPU increases the number of the plurality of arithmetic units by launching a pre-prepared program.

3. The computer device of claim 1, wherein the second device acquires the type of the new processing to be performed, references a table showing the relationship between the type of processing and the number of arithmetic units required for that type based on the acquired type, acquires the number of arithmetic units corresponding to the acquired type, and increases the number of the multiple arithmetic units based on the acquired number.

4. The computer device according to claim 1, wherein the second device monitors the status of data processing of the first device, and increases the number of the plurality of processing elements when the status satisfies the predetermined condition.

Images