Intra-server delay control device, intra-server delay control method and program

Abstract

An intra-server delay control device (100) is provided with: a sleep management unit (130) that puts a thread into a sleep when no packet arrives for a prescribed period of time, and then releases this thread from the sleep by means of a hardware interrupt when a packet arrives; a HW interrupt count management table (150a) that stores the number of hardware interrupts; and a HW interrupt frequency control unit (150) that calculates a HW interrupt frequency on the basis of the number of hardware interrupts, and controls, on the basis of the calculated HW interrupt frequency, the HW interrupt permission or prohibition associated with the sleep caused by the sleep management unit (130).

Description

[Technical field]

[0001] The present invention relates to an intra-server delay control device, an intra-server delay control method, and a program. [Background technology]

[0002] With the advancement of virtualization technology such as NFV (Network Functions Virtualization), systems are being built and operated for each service.In addition, a form called Service Function Chaining (SFC) is becoming mainstream, in which service functions are divided into reusable modules and run on independent virtual machine (VM: Virtual Machine, container, etc.) environments, allowing them to be used as components when needed and improving operability, as opposed to the form in which systems are built for each service.

[0003] A known technology for configuring virtual machines is a hypervisor environment consisting of Linux (registered trademark) and KVM (kernel-based virtual machine). In this environment, a Host OS (an OS installed on a physical server is called a Host OS) with a built-in KVM module operates as a hypervisor in a memory area called kernel space, which is different from the user space. In this environment, a virtual machine operates in the user space, and a Guest OS (an OS installed on a virtual machine is called a Guest OS) operates within the virtual machine.

[0004] A virtual machine running a Guest OS is different from a physical server running a Host OS in that all HW (hardware), including network devices (typically Ethernet card devices), is subject to register control required for interrupt processing from the HW to the Guest OS and writing from the Guest OS to the hardware. In this type of register control, notifications and processing that should be performed by physical hardware are simulated by software, so performance is generally lower than in a Host OS environment.

[0005] To address this performance degradation, there is a technology that reduces HW emulation, particularly from the Guest OS to the Host OS and external processes that exist outside the virtual machine, and improves communication performance and versatility through a high-speed, unified interface. One such technology is a device abstraction technology called Virtio, or paravirtualization technology, which has already been incorporated into many general-purpose operating systems, including Linux (registered trademark) and FreeBSD (registered trademark), and is currently in use.

[0006] Virtio defines data exchange by queues designed with ring buffers as a one-way transport for data input / output such as console, file input / output, and network communication, by queue operations. By using the Virtio queue specifications to prepare the number and size of queues appropriate for each device when the Guest OS starts, communication between the Guest OS and the outside of the virtual machine can be achieved by queue operations alone, without performing hardware emulation.

[0007] [Packet forwarding using the polling model (DPDK example)] The method of connecting and coordinating multiple virtual machines is called Inter-VM Communication, and in large-scale environments such as data centers, virtual switches have been used as a standard method for connecting VMs. However, because this method has a large communication delay, new faster methods have been proposed. For example, a method that uses special hardware called SR-IOV (Single Root I / O Virtualization) and a software method that uses Intel DPDK (Intel Data Plane Development Kit) (hereinafter referred to as DPDK), a high-speed packet processing library, have been proposed.

[0008] DPDK is a framework for controlling NICs (Network Interface Cards), which was previously handled by the Linux kernel (registered trademark), in user space. The biggest difference with the processing in the Linux kernel is that it has a polling-based reception mechanism called PMD (Pull Mode Driver). Normally, in the Linux kernel, an interrupt occurs when data arrives at the NIC, which triggers the reception process. On the other hand, in PMD, a dedicated thread continuously checks whether data has arrived and performs the reception process. By eliminating overhead such as context switches and interrupts, high-speed packet processing can be achieved. DPDK significantly improves the performance and throughput of packet processing, making it possible to secure more time for data plane application processing.

[0009] DPDK exclusively uses computer resources such as CPU (Central Processing Unit) and NIC. For this reason, it is difficult to apply to applications that require flexible reconnection on a module basis, like SFC. There is an application called SPP (Soft Patch Panel) that alleviates this problem. SPP provides shared memory between VMs and configures each VM to directly reference the same memory space, thereby eliminating packet copying in the virtualization layer. In addition, DPDK is used to speed up packet exchange between physical NICs and shared memory. SPP can change the input and output destinations of packets in software by controlling the reference destination of memory exchange for each VM. Through this process, SPP realizes dynamic connection switching between VMs and between VMs and physical NICs.

[0010] [Rx side packet processing using New API (NAPI)] FIG. 13 is a schematic diagram of Rx-side packet processing by New API (NAPI) implemented in Linux kernel 2.5 / 2.6 and later (see Non-Patent Document 1). As shown in FIG. 13, the New API (NAPI) executes a packet processing APL1 placed in a user space 60 available to a user on a server having an OS70 (e.g., a host OS), and transfers packets between the packet processing APL1 and a NIC 11 of HW 10 connected to the OS70.

[0011] The OS 70 includes a kernel 71 , a ring buffer 72 , and a driver 73 , and the kernel 71 includes a protocol processing unit 74 . Kernel 71 is a core function of OS 70 (for example, a host OS) and manages hardware monitoring and program execution status on a process-by-process basis. Here, kernel 71 responds to requests from packet processing APL1 and conveys requests from HW 10 to packet processing APL1. Kernel 71 processes requests from packet processing APL1 via a system call (a "user program running in non-privileged mode" requests processing from the "kernel running in privileged mode"). The Kernel 71 transmits a packet to the packet processing APL 1 via the Socket 75. The Kernel 71 receives a packet from the packet processing APL 1 via the Socket 75.

[0012] The ring buffer 72 is located in the memory space of the server and is managed by the Kernel 71. The ring buffer 72 is a buffer of a certain size that stores messages output by the Kernel 71 as logs, and when the upper limit size is exceeded, the messages are overwritten from the top.

[0013] Driver73 is a device driver that monitors hardware with kernel71. Driver73 depends on kernel71, and if the created (built) kernel source changes, it will become a different driver. In this case, you will need to obtain the driver source, rebuild it on the OS that uses the driver, and create the driver.

[0014] The protocol processing unit 74 performs protocol processing of L2 (data link layer), L3 (network layer), and L4 (transport layer) defined by the OSI (Open Systems Interconnection) reference model.

[0015] Socket75 is an interface that kernel71 uses for inter-process communication. Socket75 has a socket buffer and does not require frequent data copying. The flow of establishing communication via Socket75 is as follows: 1. The server creates a socket file that accepts clients. 2. The acceptance socket file is named. 3. A socket queue is created. 4. The first connection from a client in the socket queue is accepted. 5. A socket file is created on the client side. 6. The client issues a connection request to the server. 7. On the server side, a connection socket file is created in addition to the acceptance socket file. As a result of establishing communication, packet processing APL1 becomes able to call system calls such as read() and write() on kernel71.

[0016] In the above configuration, the Kernel 71 receives notification of the arrival of a packet from the NIC 11 via a hardware interrupt (hardIRQ), and schedules a software interrupt (softIRQ) for packet processing. The New API (NAPI) implemented in Linux kernel 2.5 / 2.6 above processes packets by a hardware interrupt (hardIRQ) and then a software interrupt (softIRQ) when a packet arrives. As shown in Figure 13, packet transfer using the interrupt model transfers packets using interrupt processing (see symbol a in Figure 13), which creates a wait for the interrupt processing and increases the delay in packet transfer.

[0017] The following provides an overview of packet processing on the NAPI Rx side. [Rx side packet processing configuration using New API (NAPI)] FIG. 14 is a diagram for explaining an outline of Rx-side packet processing by New API (NAPI) in the area surrounded by the dashed line in FIG. <Device driver> As shown in Figure 14, the device driver includes NIC11 (physical NIC), which is a network interface card, hardIRQ81, which is a handler that is called when a processing request is made to NIC11 and executes the requested processing (hardware interrupt), and netif_rx82, which is a processing function for software interrupts.

[0018] <Networking layer> In the networking layer, there are disposed softIRQ 83, which is a handler that is called when a processing request from netif_rx 82 occurs and executes the requested processing (software interrupt), and do_softirq 84, which is a control function unit that executes the software interrupt (softIRQ).In addition, there are disposed net_rx_action 85, which is a packet processing function unit that receives and executes a software interrupt (softIRQ), poll_list 86, which registers information on the net device (net_device) that indicates which device the hardware interrupt from the NIC 11 is from, netif_receive_skb 87, which creates an sk_buff structure (a structure that enables the Kernel 71 to recognize the state of the packet), and the Ring Buffer 72.

[0019] <Protocol layer> In the protocol layer, packet processing functional units such as ip_rcv88 and arp_rcv89 are arranged.

[0020] The above netif_rx82, do_softirq84, net_rx_action85, netif_receive_skb87, ip_rcv88, and arp_rcv89 are program components (function names) used for packet processing in Kernel71.

[0021] [Rx side packet processing operation using New API (NAPI)] The arrows (symbols) b to m in FIG. 14 indicate the flow of packet processing on the Rx side. When the hardware function unit 11a of the NIC 11 (hereinafter referred to as NIC 11) receives a packet (or frame) in a frame from the other device, it copies the arriving packet to the Ring Buffer 72 by DMA (Direct Memory Access) transfer without using the CPU (see symbol b in FIG. 14). This Ring Buffer 72 is a memory space in the server, and is managed by the Kernel 71 (see FIG. 13).

[0022] However, if the NIC 11 simply copies an arriving packet to the Ring Buffer 72, the Kernel 71 cannot recognize the packet. Therefore, when a packet arrives, the NIC 11 raises a hardware interrupt (hardIRQ) to hardIRQ 81 (see symbol c in FIG. 14), and netif_rx 82 executes the following process, which allows the Kernel 71 to recognize the packet. Note that hardIRQ 81, shown in an oval in FIG. 14, indicates a handler rather than a functional unit.

[0023] Netif_rx82 is a function that actually performs processing. When hardIRQ81 (handler) is started (see symbol d in FIG. 14), it saves net device (net_device) information, which is one of the pieces of information about the contents of the hardware interrupt (hardIRQ), in poll_list86, indicating which device the hardware interrupt from NIC11 is from. Then, netif_rx82 registers queue pruning (referring to the contents of the packets stored in the buffer, and deleting the corresponding queue entry from the buffer, taking into account the processing of the packet and the next processing to be performed) (see symbol e in FIG. 14). Specifically, when packets are stuffed into the Ring Buffer72, netif_rx82 uses the driver of NIC11 to register future queue pruning in poll_list86 (see symbol e in FIG. 14). As a result, queue pruning information resulting from the packing of packets into the Ring Buffer72 is registered in poll_list86.

[0024] In this way, in Fig.<Device driver> In the above, when the NIC 11 receives a packet, it uses DMA transfer to copy the packet that has arrived to the Ring Buffer 72. In addition, the NIC 11 raises the hardIRQ 81 (handler), and the netif_rx 82 registers the net_device in the poll_list 86 and schedules a software interrupt (softIRQ). So far, we have completed the steps in Figure 14.<Device driver> Processing of hardware interrupts in stops.

[0025] Thereafter, netif_rx82 uses the information (specifically, the pointer) in the queue stored in poll_list86 to notify softIRQ83 (handler) of the reaping of the data stored in Ring Buffer72 via a software interrupt (softIRQ) (see symbol f in Figure 14), and notifies do_softirq84, which is the software interrupt control function unit (see symbol g in Figure 14).

[0026] do_softirq84 is a software interrupt control function section, which defines each function of software interrupt (there are various types of packet processing, and interrupt processing is one of them. This defines interrupt processing). Based on this definition, do_softirq84 notifies net_rx_action85, which actually processes the software interrupt, of the current (corresponding) software interrupt request (see symbol h in Figure 14).

[0027] When the turn of the softIRQ comes, net_rx_action 85 calls a polling routine for reaping packets from Ring Buffer 72 based on net_device registered in poll_list 86 (see symbol i in FIG. 14), and reaps the packets (see symbol j in FIG. 14). At this time, net_rx_action 85 continues reaping until poll_list 86 becomes empty. Thereafter, net_rx_action 85 notifies netif_receive_skb 87 (see reference symbol k in FIG. 14).

[0028] The netif_receive_skb 87 creates an sk_buff structure, analyzes the contents of the packet, and passes the processing to the downstream protocol processor 74 (see FIG. 13) for each type. That is, the netif_receive_skb 87 analyzes the contents of the packet, and when processing is to be performed according to the contents of the packet,<Protocol layer> If it is L2, the process is passed to arp_rcv 89 (symbol m in FIG. 14).

[0029] Patent Document 1 describes a network delay control device in a server (KBP: Kernel Busy Poll). KBP constantly monitors packet arrivals using a polling model in the kernel. This suppresses softIRQ and achieves low-latency packet processing. [Prior art documents] [Patent documents]

[0030] [Patent Document 1] International Publication No. 2021 / 130828 [Non-patent literature]

[0031] [Non-Patent Document 1] New API (NAPI), [online], [searched on January 11, 2022], Internet <URL: http: / / http: / / lwn.net / 2002 / 0321 / a / napi-howto.php3> Summary of the Invention [Problem to be solved by the invention]

[0032] However, both packet transfer using the interrupt model and the polling model have the following problems. In the interrupt model, packets are transferred by software interrupt processing, which allows the kernel to process packets after receiving an event (hardware interrupt) from HW. Because the interrupt model transfers packets by interrupt (software interrupt) processing, there are issues with conflicts with other interrupts and waiting when the interrupt destination CPU is used by a higher priority process, resulting in large delays in packet transfer. In this case, if the interrupt processing becomes congested, the waiting delays will become even larger. For example, in packet transfer based on the interrupt model, packets are transferred by interrupt processing, so that waiting for the interrupt processing occurs, resulting in a large delay in packet transfer.

[0033] The mechanism by which delay occurs in the interrupt model will be explained below. In a typical kernel, packet transfer processing is performed by hardware interrupt processing and then software interrupt processing. When a software interrupt occurs in packet forwarding processing, the software interrupt processing cannot be executed immediately under the following conditions (1) to (3). Therefore, the software interrupt processing is scheduled by arbitration using a scheduler such as ksoftirqd (a kernel thread for each CPU that is executed when the load of the software interrupt becomes high), resulting in a wait on the order of ms. (1) When there is a conflict with other hardware interrupt processing (2) When there is a conflict with other software interrupt processing (3) When other high-priority processes, kernel threads (migration threads, etc.), or interrupt destination CPUs are in use Under the above conditions, the software interrupt process cannot be executed immediately.

[0034] Similarly, in packet processing using the New API (NAPI), as shown in the dotted-line box n in FIG. 14, a network delay on the order of ms occurs due to contention in interrupt processing (softIRQ). On the other hand, by using the technology described in Patent Document 1, it is possible to suppress software interrupts and realize low-latency packet harvesting by constantly monitoring packet arrival. However, since a CPU core is exclusively used and CPU time is used to monitor packet arrival, power consumption increases. In other words, the kernel thread that constantly monitors packet arrival occupies a CPU core and constantly uses CPU time, which is a problem in that power consumption increases. The relationship between the workload and the CPU utilization rate will be described with reference to Figs. 15 and 16.

[0035] Figure 15 shows an example of video (30 FPS) data transfer. The workload shown in Figure 15 has a transfer rate of 350 Mbps, and data is transferred intermittently every 30 ms.

[0036] FIG. 16 is a diagram showing the CPU utilization rate used by the busy poll thread in the KBP described in Patent Document 1. In FIG. As shown in Fig. 16, in KBP, the kernel thread occupies a CPU core to perform busy poll. Even in the case of intermittent packet reception as shown in Fig. 15, KBP has a problem of high power consumption because the CPU is always used regardless of whether packets arrive or not.

[0037] The present invention has been made in consideration of this background, and has an objective of suppressing excessive occurrence of HW interrupts caused by putting the polling thread to sleep, thereby reducing power consumption and minimizing delays within the server while transferring packets. [Means for solving the problem]

[0038] In order to solve the above-mentioned problems, the server-internal delay control device is arranged in the kernel space of an OS and launches a thread that monitors the arrival of packets using a polling model, and is characterized in that it comprises a packet arrival monitoring unit that monitors a poll list that registers information on network devices indicating which device a hardware interrupt from an interface unit is for, a packet reaping unit that, if a packet has arrived, refers to the packet stored in a ring buffer and performs reaping to delete the corresponding queue entry from the ring buffer based on the next processing to be performed, a sleep management unit that puts the thread to sleep if a packet does not arrive for a specified period of time and wakes up the thread from sleep by a hardware interrupt when a packet arrives, a HW interrupt count memory unit that stores the number of hardware interrupts, and a HW interrupt frequency control unit that calculates the HW interrupt frequency based on the number of hardware interrupts and controls whether to allow or prohibit HW interrupts by putting the sleep management unit to sleep based on the calculated HW interrupt frequency. Effect of the Invention

[0039] According to the present invention, it is possible to suppress the excessive occurrence of HW interrupts caused by putting a polling thread to sleep, reduce power consumption, and transfer packets with reduced delays within the server. [Brief description of the drawings]

[0040] [Figure 1] 1 is a schematic configuration diagram of an intra-server delay control system according to an embodiment of the present invention. [Diagram 2] This is an example of a configuration in which the polling thread (internal server delay control device) in Figure 1 is placed in the kernel space. [Diagram 3] This is an example of a configuration in which the polling thread (internal server delay control device) in Figure 1 is placed in the user space. [Figure 4]FIG. 2 is a diagram illustrating an example of the operation of a polling thread of an intra-server delay control device of an intra-server delay control system according to an embodiment of the present invention. [Diagram 5] 1 is a graph showing the relationship between the packet arrival rate and the number of hardIRQs in the server delay control system according to an embodiment of the present invention. [Figure 6] FIG. 13 is a diagram showing a HW interrupt caused by a packet arrival in a comparative example, and a polling thread started by the HW interrupt. [Figure 7] This figure shows a HW interrupt caused by a packet arrival and a polling thread started by the HW interrupt. [Figure 8] 4 is a flowchart showing NIC and HW interrupt processing of a polling thread (intra-server delay control device) of the intra-server delay control system according to the embodiment of the present invention. [Figure 9] 4 is a flowchart showing a HW interrupt frequency control process of an intra-server delay control device of the intra-server delay control system according to the embodiment of the present invention. [Figure 10] 1 is a hardware configuration diagram showing an example of a computer that realizes the functions of an intra-server delay control device of an intra-server delay control system according to an embodiment of the present invention. FIG. [Figure 11] FIG. 1 is a diagram illustrating an example in which an in-server delay control system is applied to an interrupt model in a server virtualization environment having a generic Linux kernel (registered trademark) and a VM configuration. [Figure 12] FIG. 13 is a diagram illustrating an example of applying an intra-server delay control system to an interrupt model in a server virtualization environment with a container configuration. [Figure 13] This is a schematic diagram of Rx-side packet processing using the New API (NAPI) implemented in the Linux kernel 2.5 / 2.6. [Figure 14] 14 is a diagram for explaining an outline of Rx-side packet processing by New API (NAPI) in the area surrounded by a dashed line in FIG. 13. FIG. [Figure 15] FIG. 1 is a diagram showing an example of video (30 FPS) data transfer. [Figure 16]FIG. 1 is a diagram showing the CPU utilization rate used by a busy poll thread in the KBP described in Patent Document 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Hereinafter, an intra-server delay control system and the like in an embodiment for carrying out the present invention (hereinafter, referred to as "the present embodiment") will be described with reference to the drawings. [overview] Fig. 1 is a schematic diagram of a server delay control system according to an embodiment of the present invention. This embodiment is an example of application to Rx-side packet processing using New API (NAPI) implemented in Linux kernel 2.5 / 2.6. The same components as those in Fig. 13 are given the same reference numerals. As shown in FIG. 1, the intra-server delay control system 1000 executes a packet processing APL1 placed in a user space available to a user on a server equipped with an OS (e.g., a host OS), and transfers packets between a NIC 11 of HW connected to the OS and the packet processing APL1.

[0042] The server delay control system 1000 comprises a NIC 11 (physical NIC) which is a network interface card, hardIRQ 81 which is a handler which is called when a processing request is made to NIC 11 and which executes the requested processing (hardware interrupt), a HW interrupt processing unit 182 which is a processing function unit for the HW interrupt, a receive list 186, a Ring_Buffer 72, a polling thread (server delay control device 100), and a protocol processing unit 74. The Ring Buffer 72 is a memory space in the server that is managed by the kernel. The Ring Buffer 72 is a buffer of a certain size that stores messages output by the kernel as logs, and when the upper limit size is exceeded, the messages are overwritten from the top. The protocol processing unit 74 is Ethernet, IP, TCP / UDP, etc. The protocol processing unit 74 performs protocol processing of L2 / L3 / L4 defined by the OSI reference model, for example.

[0043] <Internal server delay control device> The intra-server delay control device 100 is a polling thread located in either the kernel space or the user space. The server delay control device 100 includes a packet arrival monitoring unit 110, a packet harvesting unit 120, a sleep management unit 130, a CPU frequency / CPU idle setting unit 140, and a HW interrupt frequency control unit 150 having a HW interrupt count management table 150a (HW interrupt count memory unit).

[0044] The packet arrival monitor 110 is a thread for monitoring whether a packet has arrived. The packet arrival monitor 110 monitors (polls) the receive list 186.

[0045] The packet arrival monitoring unit 110 acquires the pointer information indicating that a packet exists in the Ring_Buffer 72 and the net_device information from the receive list 186, and transmits the information (pointer information and net_device information) to the packet harvesting unit 120. Here, if multiple pieces of packet information exist in the receive list 186, the multiple pieces of information are transmitted.

[0046] If a packet has arrived, the packet reaper 120 refers to the packet held in the Ring Buffer 72, and executes reaping to delete the corresponding queue entry from the Ring Buffer 72 based on the next process (hereinafter, this may simply be referred to as reaping packets from the Ring Buffer 72). The packet reaper 120 extracts the packet from the Ring Buffer 72 based on the received information, and transmits the packet to the protocol processor 74. When multiple packets are stored in the Ring_Buffer 72, the packet reaping unit 120 harvests the multiple packets together and passes them to the subsequent protocol processing unit 74. The number of packets harvested together is called a quota, and is often called batch processing. The protocol processing unit 74 also processes multiple packets together, making protocol processing fast.

[0047] The sleep management unit 130 puts a thread (polling thread) to sleep if no packet arrives for a predetermined period of time, and wakes up the thread (polling thread) from sleep mode using a hardware interrupt (hardIRQ) when a packet arrives (details will be described later).

[0048] During sleep, the CPU frequency / CPU idle setting unit 140 sets the CPU operating frequency of the CPU core used by the thread (polling thread) to a low value. During sleep, the CPU frequency / CPU idle setting unit 140 sets the CPU idle state of the CPU core used by this thread (polling thread) to a power saving mode (described in detail later).

[0049] The HW interrupt frequency control unit 150 calculates the HW interrupt frequency based on the number of HW interrupts, and based on the calculated HW interrupt frequency, controls whether to permit or prohibit HW interrupts through sleep by the sleep management unit 130. Specifically, the HW interrupt frequency control unit 150 compares the calculated HW interrupt frequency with a predetermined threshold, and when the HW interrupt frequency is smaller than the predetermined threshold, does not allow the sleep management unit 130 to execute sleep for a certain period of time (details will be described later).

[0050] The HW interrupt count management table 150a stores, as the number of HW interrupts, timestamps at which HW interrupts occurred (timestamps indicating when the HW interrupts occurred). The HW interrupt frequency control unit 150 calculates the number of HW interrupts based on the number of times of the timestamps stored in the HW interrupt count management table 150a.

[0051] <Location of delay control device in server> 2 and 3 are diagrams for explaining the arrangement of the polling thread (internal server delay control device 100) in FIG. -Kernel space allocation for polling threads FIG. 2 shows an example of a configuration in which the polling thread (internal server delay control device 100) in FIG. 1 is placed in the kernel space. 2, a polling thread (intra-server delay control device 100) and a protocol processing unit 74 are placed in the kernel space. This polling thread (intra-server delay control device 100) runs in the kernel space. The intra-server delay control system 1000 executes a packet processing APL1 placed in a user space on a server equipped with an OS, and transfers packets between a NIC 11 of HW and the packet processing APL1 via a device driver connected to the OS. As shown in FIG. 2, the Device driver includes a hardIRQ 81, a HW interrupt processing unit 182, a receive list 186, and a Ring_Buffer 72. A device driver is a driver for monitoring hardware.

[0052] The present invention can be applied when there is a polling thread inside the kernel, such as NAPI or KBP.

[0053] User space placement of polling threads FIG. 3 shows an example of a configuration in which the polling thread (internal server delay control device 100) in FIG. 1 is placed in the user space. 3, a polling thread (intra-server delay control device 100) and a protocol processor 74 are placed in a user space. This polling thread (intra-server delay control device 100) runs in the user space, not in the kernel space. In the intra-server delay control system 1000 shown in FIG. 3, a polling thread (intra-server delay control device 100) bypasses the kernel space and transfers packets between the device driver and the NIC 11 and the packet processing APL 1.

[0054] The present invention can be applied when there is a polling thread in user space, such as in DPDK.

[0055] The operation of the intra-server delay control system 1000 will now be described. [Rx side packet processing operation according to the present invention] Arrows (symbols) aa to jj in FIGS. 1 to 3 indicate the flow of packet processing on the Rx side. When the NIC 11 receives a packet (or a frame) from the other device, it copies the packet to the Ring Buffer 72 by DMA transfer without using the CPU (see symbols aa in Figures 1 to 3).<Device driver> It is managed by.

[0056] When a packet arrives, the NIC 11 raises a hardware interrupt (hardIRQ) to the hardIRQ 81 (handler) (see reference symbol bb in FIGS. 1 to 3), and the HW interrupt processing unit 182 executes the following process to recognize the packet.

[0057] When the hardwire 81 (handler) is started (see symbol cc in FIGS. 1 to 3), the HW interrupt processing unit 182 saves, in the receive list 186, information on the net device (net_device) that indicates which device the hardware interrupt from the NIC 11 belongs to, which is one piece of information on the contents of the hardware interrupt (hardIRQ), and registers queue pruning information. Specifically, upon receipt of the packets stuffed into the Ring Buffer 72, the HW interrupt processing unit 182 uses the driver of the NIC 11 to register the subsequent queue pruning in the receive list 186 (see symbol dd in FIGS. 1 to 3). As a result, the queue pruning resulting from the packets being stuffed into the Ring Buffer 72 is registered in the receive list 186.

[0058] The HW interrupt processing unit 182 registers net_device in the receive list 186, but does not schedule software interrupts (softIRQ), unlike the netif_rx 82 in Fig. 14. That is, the HW interrupt processing unit 182 differs from the netif_rx 82 in Fig. 14 in that it does not schedule software interrupts (softIRQ).

[0059] Furthermore, the HW interrupt processing unit 182 performs sleep release to wake up the sleeping polling thread (see symbol ee in FIGS. 1 to 3). So far, Figs. 1 to 3<Device driver> Processing of hardware interrupts in stops.

[0060] In this embodiment, the<Networking layer> 14. In this embodiment, softIRQ 83 and do_softirq 84 are deleted, and accordingly, netif_rx 82 shown in FIG. 14 does not notify softIRQ 83 (handler) to start up (see symbol f in FIG. 14).

[0061] In this embodiment, the server delay control system 1000 deletes softIRQ83 and do_softirq84 shown in FIG. 14 and instead<kernel space> A polling thread (internal delay control device 100) is provided in the internal server (see FIG. 2). Alternatively, the internal server delay control system 1000 may be configured as follows:<User space> A polling thread (internal delay control device 100) is provided (see FIG. 3).

[0062] As shown in Figure 2<kernel space> or as shown in Figure 3<User space> In the server delay control device 100, the HW interrupt frequency control unit 150 manages the HW interrupt frequency, and controls the sleep of the polling thread and the permission / prohibition of the HW interrupt according to the HW interrupt frequency (see symbols ff in FIGS. 1 to 3).

[0063] The packet arrival monitor 110 monitors (polls) the receive list 186 (see reference character gg in FIGS. 1 to 3) and checks whether a packet has arrived.

[0064] The packet arrival monitoring unit 110 acquires the pointer information indicating that a packet exists in the Ring_Buffer 72 and the net_device information from the receive list 186, and transmits the information (pointer information and net_device information) to the packet harvesting unit 120 (see symbols hh in Figs. 1 to 3). Here, if multiple packet information exists in the receive list 186, the multiple pieces of information are transmitted.

[0065] If a packet has arrived, the packet reaper 120 of the intra-server delay control device 100 retrieves the packet from the Ring Buffer 72 (see reference character ii in FIGS. 1 to 3). The packet harvesting unit 120 extracts the packet from the Ring_Buffer 72 based on the received information, and transmits the packet to the protocol processing unit 74 (see symbols jj in FIGS. 1 to 3).

[0066] [Sleep behavior of polling thread] The intra-server delay control system 1000 stops softIRQ for packet processing, which is the main cause of network delays, and executes a polling thread in which the packet arrival monitor 110 of the intra-server delay control device 100 monitors packet arrivals. Then, the packet reaper 120 processes packets using a polling model (without softIRQ) when the packets arrive.

[0067] When a packet arrives, the hardware interrupt handler wakes up a polling thread, which avoids softIRQ contention and enables immediate packet transfer processing. In other words, by putting the packet arrival monitoring function on hold and waking it up with a hardware interrupt, it is possible to achieve lower latency than packet transfer processing using software interrupts such as NAPI.

[0068] The polling thread (intra-server delay control device 100) that monitors the arrival of packets is allowed to sleep while no packets have arrived. The polling thread (intra-server delay control device 100) goes to sleep depending on whether a packet has arrived, and when a packet has arrived, it is released from sleep by hardIRQ 81. Specifically, the sleep management unit 130 of the intra-server delay control device 100 puts the polling thread to sleep depending on whether a packet has arrived, that is, if no packet has arrived for a specified period of time. The sleep management unit 130 releases sleep by hardIRQ 81 when a packet arrives. This avoids softIRQ contention and achieves low delay.

[0069] The CPU frequency / CPU idle setting unit 140 of the intra-server delay control device 100 changes the CPU operating frequency and idle setting depending on whether a packet has arrived. Specifically, the CPU frequency / CPU idle setting unit 140 lowers the CPU frequency during sleep, and raises the CPU frequency when started up again (returning the CPU operating frequency to its original state). The CPU frequency / CPU idle setting unit 140 also changes the CPU idle setting to power saving during sleep. By lowering the CPU operating frequency during sleep and changing the CPU idle setting to power saving, power saving is also achieved.

[0070] Fig. 4 is a diagram showing an example of the operation of a polling thread in the intra-server delay control device 100. The vertical axis indicates the CPU usage rate [%] of the CPU core used by the polling thread, and the horizontal axis indicates time. Fig. 4 shows an example of the operation of a polling thread due to packet arrival corresponding to the data transfer example of a video (30 FPS) in which packets are received intermittently shown in Fig. 15. As shown in Fig. 4, the sleep management unit 130 of the intra-server delay control device 100 puts the polling thread to sleep if no packet arrives for a predetermined period of time (more specifically, if a next packet does not arrive even after a fixed value (certain period) predetermined by the maintenance / operator has elapsed since the arrival of a certain packet) (see symbol p in Fig. 4).Then, the sleep management unit 130 starts the polling thread with hard IRQ 81 when a packet arrives (see symbol q in Fig. 4). During sleep, the kernel thread does not have exclusive use of the CPU core. Therefore, in addition to the polling thread, timer interrupts for stable system operation may enter the corresponding CPU core, or a migration thread for error processing, etc. may enter the corresponding CPU core, which may cause the CPU usage of the CPU core used by the polling thread to fluctuate (see symbol r in Figure 4).

[0071] [Relationship between packet arrival rate and packet receiving rate] As described above, when there is an unreceived packet in the Ring Buffer 72, the polling thread (the intra-server delay control device 100) inhibits hardware interrupt requests (hardIRQ) and performs packet reception by polling. Here, if certain conditions are met, such as the packet arrival rate being equal to the packet reception rate, the number of hardIRQs due to packet arrival increases, and hardIRQ overhead can increase packet transfer delay time and power consumption. Here we explain hardIRQ overhead. A hardware interrupt is a process with extremely high priority, and the interrupted process must suspend processing and save any intermediate processing to memory. If the number of hardIRQs increases, packet reception processing (protocol processing, etc.) is interrupted by CPU time being taken up by the hardIRQ, reducing the efficiency of packet reception processing. Below we explain the relationship between packet arrival rate and packet reception rate.

[0072] FIG. 5 is a graph showing the relationship between the packet arrival rate (packets per second), which indicates how many packets arrive per second, and the number of hardIRQs (Number of hardIRQs). In the "sparse" case, where the frequency of packet arrival is low (see the short bidirectional arrow in the middle of Figure 5), As shown in the lower left diagram of Figure 5, when the rate at which packets are accumulated in the Ring Buffer 72 (see the white arrow t in Figure 5) and the rate at which packets are received one packet at a time from the Ring Buffer 72 (see the white arrow u in Figure 5) are equal (balanced), packets do not accumulate in the Ring Buffer 72 because the frequency of packet arrival is low, and a hardIRQ is triggered every time a packet arrives.

[0073] In the case of "dense" where the frequency of packet arrival is high (see the long double-headed arrow in the middle of Figure 5), As shown in the lower right of Figure 5, if the rate at which packets accumulate in the Ring Buffer 72 (see the white arrow v in Figure 5) is high and the rate at which multiple packets are received in batch processing from the Ring Buffer 72 once a certain amount has accumulated (see the white arrow w in Figure 5) is low, the packet arrival frequency will be high and packets will accumulate in the Ring Buffer 72, so the hardIRQ prohibition time will be long and the number of hardIRQs will be small.

[0074] As shown in the dashed ellipse s in the upper diagram of Figure 5, when the packet arrival rate and packet reception rate meet a certain condition, the number of hardIRQs increases, increasing delay time and power consumption. The case in which the packet arrival rate and packet reception rate meet the certain condition is when the rate at which packets are accumulated and the rate at which packets are received are equal, as shown in the lower left diagram of Figure 5. Because the packet arrival frequency is low, packets are not accumulated in Ring Buffer 72, and a hardIRQ is triggered every time a packet arrives, increasing delay time and power consumption.

[0075] [Additional explanation of packet arrival frequency] The packet arrival frequency will now be explained in more detail. When the frequency of packet arrival is low, there are more opportunities to sleep, which can lead to power savings. However, if the frequency of packet arrival is relatively low but a hardware interrupt occurs each time the system goes into sleep and wakes up repeatedly, a hardIRQ will be triggered each time a packet arrives, increasing the latency. This embodiment provides a technique for suppressing excessive occurrence of HW interrupts caused by putting a polling thread to sleep in a situation where the packet arrival rate and the packet reception rate are at odds with each other.

[0076] [Basic concept of polling thread (internal server delay control device 100) operation] The basic concept of the operation of the polling thread (intra-server delay control device 100) will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing a HW interrupt caused by a packet arrival in a comparative example, and a polling thread started by the HW interrupt. Fig. 7 is a diagram showing a HW interrupt caused by a packet arrival in this embodiment, and a polling thread started by the HW interrupt. In the diagram, "●" indicates a packet arrival, "↑" indicates a HW interrupt, and a shaded block indicates a polling state. In the comparative example of Figure 6, when the packet arrival rate is moderate, a large number of HW interrupts occur (see symbol x in Figure 6), and the polling thread enters the polling state due to the large number of HW interrupts. HW interrupts become excessive, and the delay time due to overhead increases. Note that when packet arrival rate is sparse or dense, the number of HW interrupts occurs is small, and the delay time due to overhead is small.

[0077] Therefore, in this embodiment, the polling thread (the intra-server delay control device 100) calculates the HW interrupt frequency f per unit time, and compares and evaluates the calculated HW interrupt frequency f with a threshold value preset by the operator. Then, a logic is introduced that puts the polling thread to sleep and determines whether to permit subsequent HW interrupts. Take the case of a medium packet arrival in FIG. 7 as an example. As indicated by the symbol y in FIG. 7, the polling thread (the intra-server delay control device 100) takes into account the number of HW interrupts per unit time and controls interrupt inhibition / permission according to the HW interrupt frequency (see the dashed box z in FIG. 7). This prevents excessive HW interrupts, and suppresses the phenomenon in which excessive HW interrupts cause an increase in delay time due to overhead.

[0078] [Operation flow of polling thread (internal delay control device 100)] FIG. 8 is a flowchart showing the NIC and HW interrupt processing of the polling thread (intra-server delay control device 100). While the polling thread is running, this operation flow is executed in a loop. This flow starts when a packet arrives at the NIC 11. In step S1, the NIC 11 copies the received packet data to a memory area by DMA (Direct Memory Access).

[0079] In step S2, the polling thread (the server delay control device 100) determines whether or not a HW interrupt is permitted. If a HW interrupt is permitted (S2: Yes), the process proceeds to step S3. If a HW interrupt is not permitted (S2: No), the process ends. In step S 3 , the NIC 11 launches a HW interrupt (hardIRQ) to the hardIRQ 81 (handler) to activate the HW interrupt, and registers packet arrival information (NIC device information, etc.) in the receive list 186 . In step S4, if the polling thread (internal delay control device 100) is sleeping, the NIC 11 wakes up the polling thread and ends the process of this flow.

[0080] FIG. 9 is a flowchart showing the HW interrupt frequency control process of the polling thread (the intra-server delay control device 100). When the polling thread is sleeping, a packet arrives, it is woken up by a HW interrupt and this flow starts. In step S11, the HW interrupt frequency control unit 150 prohibits a HW interrupt by the NIC 11. If a HW interrupt occurs during processing, the processing will be interrupted, so the HW interrupt frequency control unit 150 temporarily prohibits a HW interrupt by the NIC 11.

[0081] In step S12, the CPU frequency / CPU idle setting unit 140 sets a high CPU frequency for the CPU core in which the polling thread operates, and releases the CPU from the idle state if the CPU is in the idle state.

[0082] In step S13, the HW interrupt frequency control unit 150 records the time stamp at which the HW interrupt occurred in the HW interrupt count management table 150a.

[0083] In step S14, the polling thread refers to the receive list 186. The polling thread finds out from which device the HW interrupt has occurred, and in the next step S15, checks the packet arrival information in the receive list 186. Note that, instead of referring to a control plane list called receive list 186, it is also possible to check whether a packet has arrived by directly referring to ring buffer 72. For example, NAPI implemented in the Linux kernel monitors a control plane list called poll_list.

[0084] In step S15, the packet arrival monitor 110 determines whether or not packet arrival information exists in the receive list 186. If there is no packet arrival information in the receive list 186 (S15: No), that is, if there is no packet to be processed, the following process is skipped and the process proceeds to step S20.

[0085] If packet arrival information exists in the receive list 186 (S15: Yes), in step S16 the polling thread refers to the packet data from the ring buffer 72 and transfers the corresponding data to the subsequent protocol processing unit 74. Here, if there are multiple pieces of data, they may be received and processed in a batch.

[0086] In step S17, the packet harvester 120 determines whether or not there is an unreceived packet in the ring buffer 72. If there is an unreceived packet in the ring buffer 72 (S17: Yes), the process returns to step S16.

[0087] If there are no unreceived packets in the ring buffer 72 (S17: No), in step S18, the HW interrupt frequency control unit 150 refers to the HW interrupt count management table 150a and calculates the HW interrupt frequency f per unit time. The HW interrupt count management table 150a records timestamps at which HW interrupts have occurred. The HW interrupt frequency control unit 150 calculates the HW interrupt frequency f per unit time based on the number of timestamps recorded in the HW interrupt count management table 150a. By the way, by running a polling loop while disabling HW interrupts, the HW interrupt frequency f becomes smaller with each calculation.

[0088] In step S19, the HW interrupt frequency control unit 150 determines whether the HW interrupt frequency f is smaller than a threshold value set by an operator. If the HW interrupt frequency f is equal to or greater than the threshold value (e.g., two or more HW interrupts in 50 μs) (S19: No), the process proceeds to step S24. The determination based on the HW interrupt frequency f may not be based on a fixed threshold value set by an operator, but may be based on a threshold value that is dynamically determined by learning according to the characteristics of the incoming traffic.

[0089] If the HW interrupt frequency f is smaller than the threshold set by the operator (S19: Yes), the process proceeds to the sleep control in step S20 and thereafter. If the HW interrupt frequency f is smaller than the threshold, this corresponds to the case where the packet arrival is "sparse" as shown in FIG.

[0090] In step S20, the CPU frequency / CPU idle setting unit 140 sets the CPU frequency of the CPU core in which the polling thread operates to a low value, and puts the CPU in the idle state.

[0091] In step S 21 , the packet arrival monitor unit 110 deletes the relevant NIC information from the receive list 186 . In step S22, the HW interrupt frequency control unit 150 permits a HW interrupt from the NIC. In step S23, the sleep management unit 130 puts the polling thread to sleep, and ends the process of this flow.

[0092] On the other hand, if the HW interrupt frequency f is equal to or greater than the threshold in step S19, the process proceeds to step S24. If the HW interrupt frequency f is equal to or greater than the threshold, this corresponds to the packet arrival being in the "medium" state as shown in FIG. 7, and the execution result of step S24 corresponds to the interrupt prohibition shown in the dashed box z in FIG. In step S24, the polling thread is put to sleep for a predetermined time (for example, after a certain time t has elapsed), and the process proceeds to step S16.

[0093] The polling thread is made to sleep for the above-mentioned fixed time t. As described in the example of the "medium" packet arrival in FIG. 7, when the HW interrupt frequency f is equal to or greater than the threshold, even if the polling thread is made to sleep, a packet arrives immediately and the polling thread is woken up by the HW interrupt. In other words, if the polling thread is not sleeping, an event occurs that would not have been woken up by the HW interrupt, and excessive HW interrupts occur. Therefore, in this embodiment, when the HW interrupt frequency f is equal to or greater than the threshold, the polling thread is made to sleep after the fixed time t has elapsed. In other words, even if the condition for putting the polling thread to sleep is met, if the HW interrupt frequency f is equal to or greater than the threshold, a time is set aside before the polling thread goes to sleep (the polling thread waits until the fixed time t has elapsed before going to sleep). This makes it possible to suppress excessive HW interrupts when the packet arrival rate and the packet reception rate satisfy a certain condition (when the rate at which packets are stored in the Ring Buffer 72 is equal to the rate at which packets are received), as in the case of the "medium" packet arrival shown in FIG. 7.

[0094] Here, if the program would normally be sleeping, the polling thread can be made to sleep after a certain time t has elapsed, shortening the sleep time. However, since HW interrupts have a large delay, it is more efficient overall to allow a delay before going into sleep, considering that the program will be woken up by a HW interrupt every time.

[0095] The fixed time t may be zero. If the fixed time t is set to zero, the effect of suppressing excessive HW interrupts that is provided by this embodiment cannot be obtained, but the flow in FIG. 9 can be executed at all times. In other words, if the fixed time t is set to zero, the effect is substantially the same as when this system is not applied. This eliminates the need for a process to determine whether or not to execute the flow in FIG. 9. In other words, since it is only necessary to change the setting of t, this can be implemented in a general purpose manner without modifying existing systems.

[0096] For convenience of explanation, t is referred to as a fixed time, but t may be a variable time (predetermined time). For example, t may be determined by learning from input traffic characteristics, or the value of t may be increased little by little each time the loop of the flow in Fig. 9 is repeated (for example, every 1 μs). By gradually increasing the value of t, when the HW interrupt frequency f is equal to or greater than a threshold, the time until sleep is extended in accordance with the current state of bucket arrival, and excessive HW interrupts can be effectively suppressed.

[0097] Incidentally, according to actual measurements by the inventors, it has been confirmed that the deterioration in latency caused by executing the sleep control flow in Figure 9 is limited to an average of a few microseconds, which is a small overhead when viewed alone as being equivalent to waking up due to a hardware interrupt.

[0098] Although an example was explained in which the effect is achieved when the packet arrival rate in Fig. 7 is "medium," there is also an opportunity to sleep and wake up with a hardIRQ when the packet arrival rate in Fig. 7 is "dense." Therefore, the effect of the present invention can be expected by controlling the frequency of hardIRQ even when the packet arrival rate is "dense."

[0099] [Hardware configuration] The intra-server delay control device 100 according to the above embodiment is realized by a computer 900 having a configuration as shown in FIG. FIG. 10 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the intra-server delay control device 100. As shown in FIG. The computer 900 has a CPU 901 , a ROM 902 , a RAM 903 , a HDD 904 , a communication interface (I / F) 906 , an input / output interface (I / F) 905 , and a media interface (I / F) 907 .

[0100] The CPU 901 operates based on a program stored in the ROM 902 or HDD 904, and controls each part of the intra-server delay control device 100 shown in Figures 1 to 3. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started, programs that depend on the hardware of the computer 900, and the like.

[0101] The CPU 901 controls an input device 910 such as a mouse and a keyboard, and an output device 911 such as a display, via an input / output I / F 905. The CPU 901 acquires data from the input device 910 via the input / output I / F 905, and outputs generated data to the output device 911. Note that a GPU (Graphics Processing Unit) or the like may be used as a processor together with the CPU 901.

[0102] The HDD 904 stores programs executed by the CPU 901 and data used by the programs, etc. The communication I / F 906 receives data from other devices via a communication network (e.g., NW (Network) 920) and outputs the data to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network.

[0103] The media I / F 907 reads the program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program related to the target process from the recording medium 912 onto the RAM 903 via the media I / F 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductor memory tape medium, or a semiconductor memory, etc.

[0104] For example, when the computer 900 functions as the in-server delay control device 100 configured as one device according to this embodiment, the CPU 901 of the computer 900 realizes the functions of the in-server delay control device 100 by executing the program loaded onto the RAM 903. Also, the data in the RAM 903 is stored in the HDD 904. The CPU 901 reads and executes the program related to the target process from the recording medium 912. Additionally, the CPU 901 may read the program related to the target process from another device via the communication network (NW920).

[0105] [Application Example] It can be applied to an in-server delay control device that starts a thread for monitoring packet arrival using the polling model in the Kernel, like the polling thread (in-server delay control device 100) shown in FIG. 2. In this case, the OS is not limited. Also, it is not limited to being in a server virtualization environment. Therefore, the in-server delay control system 1000 can be applied to each configuration shown in FIGS. 11 and 12.

[0106] <Example of Application to VM Configuration> FIG. 11 is a diagram showing an example in which the in-server delay control system 1000A is applied to the interrupt model in a server virtualization environment of a general-purpose Linux kernel (registered trademark) and VM configuration. The same reference numerals are given to the same components as in FIG. 1. As shown in FIG. 11, in an intra-server delay control system 1000A, an intra-server delay control device 100 is disposed in a kernel 171 of a guest OS 70, and an intra-server delay control device 100 is disposed in a kernel 91 of a host OS 90.

[0107] In detail, the server includes a Host OS 90 on which a virtual machine and an external process formed outside the virtual machine can run, and a Guest OS 70 that runs within the virtual machine. The HostOS90 comprises a Kernel91, a Ring Buffer22 managed by the Kernel91 in the memory space of a server equipped with the HostOS90, a receive list186 (Figure 2) that registers information about network devices indicating which device the hardware interrupt (hardIRQ) from the NIC11 belongs to, a vhost-net module221 which is a kernel thread, a tap device222 which is a virtual interface created by the Kernel91, and a virtual switch (br)223.

[0108] The kernel 91 includes an intra-server delay control device 100 . The kernel 91 transmits the packet to the virtual machine 30 via the tap device 222 .

[0109] On the other hand, GuestOS70 comprises Kernel171, Ring Buffer52 managed by Kernel171 in the memory space of the server having GuestOS70, receive list186 (Figure 2) which registers information on network devices indicating which device the hardware interrupt (hardIRQ) from NIC11 belongs to, and Socket75 which is an interface for Kernel171 to perform inter-process communication.

[0110] The kernel 171 includes an intra-server delay control device 100 and a protocol processing unit 74 that performs protocol processing of the reaped packet. The kernel 171 transmits the packet to the packet processing APL 1 via the protocol processing unit 74 .

[0111] By doing so, in a system with a virtual server configuration of VMs, in both the HostOS 90 and the GuestOS 70, it is possible to perform packet transfer with reduced delay within the server without modifying the APL.

[0112] <Example of application to container configuration> 12 is a diagram showing an example in which an intra-server delay control system 1000B is applied to an interrupt model in a server virtualization environment with a container configuration. The same components as those in FIG. 1 are denoted by the same reference numerals. 12, the intra-server delay control system 1000B includes a Guest OS 180 and a container configuration in which the OS is replaced with a Container 210. The Container 210 includes a vNIC (virtual NIC) 211. The intra-server delay control device 100 is disposed in a Kernel 181 of the Guest OS 180.

[0113] In systems with virtual server configurations such as containers, packet transfer can be performed with reduced latency within the server without modifying the APL.

[0114] <Example of application to bare metal configuration (non-virtualized configuration)> The present invention can be applied to systems with a non-virtualized configuration, such as a bare metal configuration. In a system with a non-virtualized configuration, it is possible to transfer packets with reduced delays within the server without modifying the APL.

[0115] <Scale in / out> When traffic volume is high and multiple NIC devices or NIC ports are used, by running multiple polling threads in association with these, it is possible to scale in / out the polling threads while controlling the HW interrupt frequency.

[0116] <Extension Technology> When the number of traffic flows increases, the present invention works in conjunction with RSS (Receive-Side Scaling), which can process inbound network traffic using multiple CPUs, to increase the number of CPUs assigned to the packet arrival monitoring thread, making it possible to scale out in response to network load.

[0117] [effect] As described above, the intra-server delay control device 100 (see Figs. 1 and 2) is arranged in the kernel space of the OS and starts a thread for monitoring packet arrival using a polling model. The intra-server delay control device 100 comprises a packet arrival monitoring unit 110 that monitors (polls) a poll list (receive list 186) that registers information on a network device indicating which device a hardware interrupt (hardIRQ) from an interface unit (NIC 11) belongs to; a packet reaping unit 120 that, when a packet has arrived, refers to the packet held in a ring buffer (Ring Buffer 72) and executes reaping to delete the corresponding queue entry from the ring buffer based on the next process to be performed; and a polling thread that puts the polling thread to sleep if no packet has arrived for a predetermined period of time, and restarts this thread by a hardware interrupt (hardIRQ) when a packet has arrived. The system includes a sleep management unit 130 that cancels the sleep of a certain thread (a certain number of threads), a HW interrupt count storage unit (a HW interrupt count management table 150a) that stores the number of hardware interrupts, and a HW interrupt frequency control unit 150 that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit 130 based on the calculated HW interrupt frequency.

[0118] In this way, the intra-server delay control device 100 stops software interrupts (softIRQ) for packet processing, which is the main cause of NW delays, and the packet arrival monitor 110 of the intra-server delay control device 100 executes a thread for monitoring packet arrival, and the packet harvester 120 processes the packet by a polling model (without softIRQ) when a packet arrives. If no packet arrives for a predetermined period, the sleep manager 130 puts the thread to sleep, so that the thread goes to sleep when no packet arrives. The sleep manager 130 releases the sleep state by a hardware interrupt (hardIRQ) when a packet arrives. Furthermore, the HW interrupt frequency controller 150 calculates the HW interrupt frequency based on the HW interrupt count stored in the HW interrupt count storage unit (HW interrupt count management table 150a), and controls whether to permit or prohibit HW interrupts by the sleep manager 130 based on the calculated HW interrupt frequency. This provides the following advantages (1) to (4).

[0119] (1) Software interrupts (softIRQs) that cause delays when packets arrive are stopped, and a polling model is implemented in the kernel (Kernel 171). In other words, unlike the existing technology NAPI, the intra-server delay control system 1000 implements a polling model rather than an interrupt model that is the main cause of network delays. When a packet arrives, it is immediately harvested without waiting, making it possible to implement low-delay packet processing.

[0120] (2) The polling thread in the intra-server delay control device 100 operates as a kernel thread and monitors packet arrivals in polling mode. The kernel thread (polling thread) that monitors packet arrivals sleeps while no packets have arrived. When no packets have arrived, the CPU is not used due to the sleep, which results in power saving effects.

[0121] Then, when a packet arrives, the polling thread that was sleeping is woken up (woke from sleep) by the hardIRQ handler at the time of packet arrival. By waking up from sleep by the hardIRQ handler, it is possible to immediately start the polling thread while avoiding softIRQ contention. A distinctive feature of this method is that sleep is not woken up by a timer, but by the hardIRQ handler. Note that if the traffic load is known in advance, for example, if a 30 ms sleep is known as the workload transfer rate shown in Figure 15, then the hardIRQ handler may be woken up at this timing.

[0122] (3) Furthermore, based on the calculated HW interrupt frequency, the sleep management unit 130 controls whether to permit or prohibit HW interrupts. This makes it possible to suppress an increase in delay time caused by excessive HW interrupts, which is one of the overheads caused by putting a polling thread to sleep, and achieves even lower delays.

[0123] (4) The present invention can be applied to a case where there is a polling thread inside the kernel, such as NAPI or KBP.

[0124] In this way, the intra-server delay control device 100 (see FIG. 2) can achieve both low delay and power saving by managing the sleep of the polling thread that performs packet transfer processing when there is a polling thread inside the kernel. Furthermore, when a certain condition is met, such as the packet arrival rate being equal to the packet reception rate, it is possible to prevent an increase in the number of HW interrupts due to packet arrival, which would otherwise increase the packet transfer delay time.

[0125] Also, the intra-server delay control device 100 (see Figs. 1 and 3) is arranged in a user space and starts a thread for monitoring arrival of a packet using a polling model. The intra-server delay control device 100 comprises a packet arrival monitoring unit 110 that monitors (polls) a poll list (receive list 186) that registers information on a network device indicating which device a hardware interrupt (hardIRQ) from an interface unit (NIC 11) belongs to, a packet reaping unit 120 that, when a packet has arrived, refers to the packet held in a ring buffer (Ring Buffer 72) and executes reaping to delete the entry of the corresponding queue from the ring buffer based on the next process to be performed, and a polling thread that puts the polling thread to sleep if no packet has arrived for a predetermined period of time, and, when a packet has arrived, wakes up this thread by a hardware interrupt (hardIRQ). The system includes a sleep management unit 130 that cancels the sleep of a certain thread (a certain number of threads), a HW interrupt count storage unit (a HW interrupt count management table 150a) that stores the number of hardware interrupts, and a HW interrupt frequency control unit 150 that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit 130 based on the calculated HW interrupt frequency.

[0126] In this way, the intra-server delay control device 100 (see FIG. 3) can achieve both low latency and power saving by performing sleep management of the polling thread that performs packet forwarding processing when there is a polling thread in the user space, as in DPDK. Furthermore, when a certain condition is met, such as the packet arrival rate being equal to the packet reception rate, it is possible to suppress an increase in delay time caused by excessive occurrence of HW interrupts, thereby achieving even lower latency.

[0127] Furthermore, a Guest OS (Guest OS 70) running within the virtual machine has a kernel (Kernel 171), a ring buffer (Ring Buffer 72) managed by the kernel in a memory space within the server having the Guest OS, a poll list (poll_list 186) for registering information on network devices indicating which device a hardware interrupt (hardIRQ) from the interface unit (NIC 11) belongs to, and a protocol processing unit 74 for performing protocol processing of a packet for which reaping has been executed. The kernel is provided with an intra-server delay control device 100 for starting up a thread for monitoring arrival of a packet using a polling model. The intra-server delay control device 100 has a packet arrival monitoring unit 110 for monitoring (polling) the poll list, a packet reaping unit 120 for reaping a packet that has arrived and that refers to the packet held in the ring buffer and deletes the entry of the corresponding queue from the ring buffer based on the next process to be performed, and a polling thread for putting the thread to sleep if a packet has not arrived for a predetermined period of time, and for starting up the thread when a packet has arrived. The system is characterized by comprising a sleep management unit 130 which performs sleep release by a hardware interrupt (hardIRQ) of a sleep thread, a HW interrupt count storage unit (HW interrupt count management table 150a) which stores the number of hardware interrupts, and a HW interrupt frequency control unit 150 which calculates the HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit 130 based on the calculated HW interrupt frequency.

[0128] In this way, in a system with a virtual server configuration of VMs, for a server equipped with a Guest OS (Guest OS 70), it is possible to reduce power consumption and perform packet transfer with reduced delay within the server without modifying the APL. Furthermore, when a certain condition is satisfied, such as the packet arrival speed being equal to the packet reception speed, it is possible to suppress an increase in delay time caused by excessive occurrence of HW interrupts, thereby achieving even lower delay.

[0129] A Host OS (HostOS90) on which a virtual machine and an external process formed outside the virtual machine can operate includes a kernel (Kernel91), a ring buffer (Ring Buffer22) managed by the kernel in a memory space in a server having the Host OS, a poll list (poll_list186) for registering information on a network device indicating which device a hardware interrupt (hardIRQ) from an interface unit (NIC11) belongs to, and a tap device 222 which is a virtual interface created by the kernel (Kernel91). The kernel includes an intra-server delay control device 100 which starts a thread (thread) for monitoring arrival of packets using a polling model. The intra-server delay control device 100 includes a packet arrival monitoring unit 110 which monitors (polling) the poll list, a packet reaping unit 120 which, if a packet has arrived, refers to the packet held in the ring buffer (Ring Buffer72) and executes reaping to delete the entry of the corresponding queue from the ring buffer (Ring Buffer72) based on the next process, and, if a packet has not arrived for a predetermined period of time, starts a thread (polling The system is characterized by comprising a sleep management unit 130 that puts a polling thread into sleep and, when a packet arrives, wakes up this thread (polling thread) with a hardware interrupt (hardIRQ), and a HW interrupt frequency control unit 150 that calculates a HW interrupt frequency based on the number of hardware interrupts and controls whether to permit or prohibit HW interrupts due to the sleep of the sleep management unit 130 based on the calculated HW interrupt frequency.

[0130] In this way, in a system with a virtual server configuration of VMs, for a server equipped with a kernel (Kernel 171) and a Host OS (Host OS 90), it is possible to reduce power consumption and perform packet transfer with reduced delay within the server without modifying the APL. Furthermore, when a certain condition is met, such as the packet arrival speed being equal to the packet reception speed, it is possible to suppress an increase in delay time caused by excessive occurrence of HW interrupts, thereby achieving even lower delay.

[0131] In the intra-server delay control device 100, the HW interrupt frequency control unit 150 compares the calculated HW interrupt frequency with a predetermined threshold value and determines whether the HW interrupt frequency is greater than or equal to the predetermined threshold value. The above In this case, the sleep management unit 130 executes sleep for a certain period of time. R It is characterized by:

[0132] By doing this, when certain conditions are met, such as the packet arrival rate and the packet receiving rate being equal, it is possible to suppress increases in delay time caused by excessive HW interrupts, thereby achieving even lower delays.

[0133] <Other Effects> In an intra-server delay control system including an intra-server delay control device 100, an interface unit (NIC 11) stops scheduling by software interrupt when a hardware interrupt is started up in an interrupt handler (hardIRQ 81) and a device is registered in a poll list.

[0134] By doing this, softIRQ conflicts can be avoided by stopping the softIRQ that causes delays and instead processing packets in the context of the hardIRQ, which has a higher priority, when triggered by a timer.

[0135] In this embodiment, the intra-server delay control device 100 includes a CPU frequency setting unit (CPU frequency / CPU idle setting unit 140) that sets a low CPU operating frequency of a CPU core used by a thread during sleep.

[0136] In this way, the intra-server delay control device 100 dynamically varies the CPU operating frequency in accordance with the traffic; in other words, if the CPU is not used due to sleep, the CPU operating frequency during sleep can be set low, thereby further enhancing the power saving effect.

[0137] In this embodiment, the intra-server delay control device 100 includes a CPU idle setting unit (CPU frequency / CPU idle setting unit 140) that sets the CPU idle state of the CPU core used by the thread to a power saving mode during sleep.

[0138] By doing this, the server delay control device 100 can dynamically change the CPU idle state (power saving function according to the CPU model, such as changing the operating voltage) in accordance with the traffic, thereby further enhancing the power saving effect.

[0139] In addition, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0140] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized by software for a processor to interpret and execute a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC (Integrated Circuit) card, SD (Secure Digital) card, or optical disk. [Explanation of symbols]

[0141] 1 Packet processing APL (application) 10 HW 11 NIC (physical NIC) (interface part) 70 Guest OS 74 Protocol Processing Unit 60 user space 72 Ring Buffer 90 Host OS (OS) 91,171,181 Kernel 100 Server delay control device (polling thread) 110 Packet arrival monitor 120 Packet Reaper 130 sleep management department 140 CPU frequency / CPU idle setting section 150 HW interrupt frequency control unit 150a HW interrupt count management table (HW interrupt count storage unit) 180 Guest OS (OS) 186 receive list 210 Container 1000, 1000A, 1000B Server Delay Control System

Claims

1. A server-side delay control device that is placed in a kernel space of an OS and starts a thread that monitors arrival of packets using a polling model, a packet arrival monitor unit that monitors a poll list that registers information on a network device indicating which device a hardware interrupt from the interface unit belongs to; a packet reaping unit that, when a packet has arrived, refers to the packet stored in the ring buffer and executes reaping to delete the corresponding queue entry from the ring buffer based on the next process to be performed; a sleep management unit that puts the thread to sleep if no packet arrives for a predetermined period of time, and that wakes up the thread from sleep mode by a hardware interrupt when a packet arrives; a hardware interrupt count storage unit that stores a hardware interrupt count; a HW interrupt frequency control unit that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit based on the calculated HW interrupt frequency. The delay control device in a server is characterized by:

2. A server-side delay control device that is located in a user space and launches a thread that monitors packet arrivals using a polling model, a packet arrival monitor unit that monitors a poll list that registers information on a network device indicating which device a hardware interrupt from the interface unit belongs to; a packet reaping unit that, when a packet has arrived, refers to the packet stored in the ring buffer and executes reaping to delete the corresponding queue entry from the ring buffer based on the next process to be performed; a sleep management unit that puts the thread to sleep if no packet arrives for a predetermined period of time, and that wakes up the thread from sleep mode by a hardware interrupt when a packet arrives; a hardware interrupt count storage unit that stores a hardware interrupt count; a HW interrupt frequency control unit that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit based on the calculated HW interrupt frequency. The delay control device in a server is characterized by:

3. A server-internal delay control device, The guest OS running in the virtual machine The kernel, a ring buffer managed by the kernel in a memory space in the server having the Guest OS; a poll list for registering information on a network device that indicates which device a hardware interrupt from the interface unit belongs to; a protocol processing unit that performs protocol processing of the reaped packet, The server delay control device includes a thread in the kernel that uses a polling model to monitor packet arrivals, The server internal delay control device includes: a packet arrival monitor unit that monitors arrival of packets using the poll list; a packet reaping unit that, when a packet has arrived, refers to the packet stored in the ring buffer and executes reaping to delete the corresponding queue entry from the ring buffer based on the next process to be performed; a sleep management unit that puts the thread to sleep if no packet arrives for a predetermined period of time, and that wakes up the thread from sleep mode by a hardware interrupt when a packet arrives; a hardware interrupt count storage unit that stores a hardware interrupt count; a HW interrupt frequency control unit that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit based on the calculated HW interrupt frequency. The delay control device in a server is characterized by:

4. A server-internal delay control device, A host OS in which a virtual machine and an external process formed outside the virtual machine can operate, The kernel, a ring buffer managed by the kernel in a memory space in the server having the Host OS; a poll list for registering information on a network device that indicates which device a hardware interrupt from the interface unit belongs to; a tap device that is a virtual interface created by the kernel; The server delay control device includes a thread in the kernel that uses a polling model to monitor packet arrivals, The server internal delay control device includes: a packet arrival monitor unit that monitors arrival of packets using the poll list; a packet reaping unit that, when a packet has arrived, refers to the packet stored in the ring buffer and executes reaping to delete the corresponding queue entry from the ring buffer based on the next process to be performed; a sleep management unit that puts the thread to sleep if no packet arrives for a predetermined period of time, and that wakes up the thread from sleep mode by a hardware interrupt when a packet arrives; a hardware interrupt count storage unit that stores a hardware interrupt count; a HW interrupt frequency control unit that calculates a HW interrupt frequency based on the number of hardware interrupts and controls permission or prohibition of HW interrupts due to sleep of the sleep management unit based on the calculated HW interrupt frequency. The delay control device in a server is characterized by:

5. The HW interrupt frequency control unit compares the calculated HW interrupt frequency with a predetermined threshold, and when the HW interrupt frequency is equal to or greater than the predetermined threshold, causes the sleep management unit to execute sleep for a certain period of time. The server delay control device according to any one of claims 1 to 4.

6. A method for controlling delay in a server by a delay control device in a server, the method including: starting a thread that is placed in a kernel space of an OS and monitors arrival of a packet using a polling model; The server delay control device includes: A step of monitoring a poll list that registers information on a network device indicating which device a hardware interrupt from the interface unit belongs to; If a packet has arrived, a pruning step is performed to refer to the packet stored in the ring buffer and delete the corresponding queue entry from the ring buffer based on the next process to be performed; a sleep management step of putting the thread to sleep if no packet arrives for a predetermined period of time, and waking up the thread from sleep mode by a hardware interrupt when a packet arrives; storing a hardware interrupt count; a step of calculating a HW interrupt frequency based on the number of hardware interrupts, and controlling permission or prohibition of HW interrupts by sleep in the sleep management step based on the calculated HW interrupt frequency. A method for controlling delay within a server.

7. A program for causing a computer to function as the server internal delay control device according to any one of claims 1 to 5.

Images