Server-based delay control device, Server-based delay control method, and program

Abstract

The challenge is to reduce power consumption by suppressing excessive hardware interrupts through the sleep function of the polling thread, while also minimizing latency within the server and enabling packet forwarding. [Solution] The server-internal delay control device 100 includes a packet harvesting unit 120 that, when a packet has arrived, refers to the packet held in the ring buffer 72 and performs harvesting by deleting the corresponding queue entry from the ring buffer; a sleep management unit 130 that puts a thread to sleep and wakes the thread from sleep by a hardware interrupt when a packet arrives; and a management unit 160 that, when the timing of data arrival can be predicted, performs pre-wake control to wake the thread immediately before the arrival of the data in accordance with the timing of the data arrival.

Description

Technical Field

[0001] The present invention relates to an in-server delay control device, an in-server delay control method, and a program.

Background Art

[0002] Based on the progress of virtualization technologies such as NFV (Network Functions Virtualization), systems are being constructed and operated for each service. Also, from the form of constructing a system for each service, the service functions are divided into reusable module units and operated on an independent virtual machine (VM: Virtual Machine or container, etc.) environment, and a form called SFC (Service Function Chaining) that enhances operability by using it as needed like a component is becoming the mainstream.

[0003] As a technology for constructing a virtual machine, a hypervisor environment composed of Linux (registered trademark) and KVM (kernel-based virtual machine) is known. In this environment, a Host OS (the OS installed on a physical server is called a Host OS) incorporating a KVM module operates in a memory area different from the user space called the kernel space as a hypervisor. In this environment, a virtual machine operates in the user space, and a Guest OS (the OS installed on the virtual machine is called a Guest OS) operates within the virtual machine.

[0004] Unlike a physical server running a Host OS, a virtual machine running a Guest OS uses all hardware (including network devices such as Ethernet card devices) for register control necessary for interrupt handling from the hardware to the Guest OS and writing from the Guest OS to the hardware. In such register control, notifications and processes that should be performed by the physical hardware are simulated by software, so performance is generally lower than in a Host OS environment.

[0005] In addressing this performance degradation, particularly from the Guest OS to the Host OS and external processes outside the virtual machine, there are technologies that reduce hardware imitation and improve communication performance and versatility through a fast and unified interface. One such technology is virtio, a device abstraction technology, or paravirtualization technology, which has already been developed and is incorporated into many general-purpose operating systems, including Linux and FreeBSD (registered trademark), and is currently in use.

[0006] In virtio, data exchange for data input / output such as console, file input / output, and network communication is defined by queue operations, using queues designed with ring buffers as a transport for unidirectional data transfer. By utilizing the virtio queue specifications and preparing the appropriate number and size of queues for each device at the time of Guest OS startup, communication between the Guest OS and the outside of the virtual machine can be achieved solely through queue operations without performing hardware emulation.

[0007] [Packet forwarding using a polling model (DPDK example)] The method of connecting and coordinating multiple virtual machines is called Inter-VM Communication, and in large-scale environments such as data centers, virtual switches have been the standard for connecting VMs. However, because this method has significant communication latency, faster methods are being proposed. For example, methods using special hardware called SR-IOV (Single Root I / O Virtualization) and software-based methods using Intel DPDK (Intel Data Plane Development Kit) (hereinafter referred to as DPDK), a high-speed packet processing library, have been proposed.

[0008] DPDK is a framework for controlling NICs (Network Interface Cards) in user space, a task traditionally handled by the Linux kernel. The biggest difference from the Linux kernel's processing is its polling-based receiving mechanism called PMD (Pull Mode Driver). Normally, the Linux kernel triggers an interrupt upon data arrival at the NIC, which then initiates the receiving process. In contrast, PMD uses a dedicated thread to continuously check for data arrival and perform receiving processing. By eliminating overhead such as context switching and interrupts, it enables high-speed packet processing. DPDK significantly improves packet processing performance and throughput, freeing up more time for data plane application processing.

[0009] DPDK exclusively uses computer resources such as the CPU (Central Processing Unit) and NIC. Therefore, it is difficult to apply to applications that require flexible module-based switching, such as SFCs. SPP (Soft Patch Panel) is an application that mitigates this limitation. SPP eliminates packet copying at the virtualization layer by providing shared memory between VMs, allowing each VM to directly access the same memory space. Furthermore, it uses DPDK to accelerate packet exchange between the physical NIC and shared memory. SPP can software-change the input and output destinations of packets by controlling the memory exchange references of each VM. Through this process, SPP enables dynamic connection switching between VMs and between VMs and physical NICs.

[0010] [Rx-side packet processing using the New API (NAPI)] Figure 22 is a schematic diagram of Rx-side packet processing using the New API (NAPI) implemented in Linux kernel 2.5 / 2.6 (see Non-Patent Document 1). As shown in Figure 22, the New API (NAPI) executes a packet processing APL1 located in the user space 60 accessible to the user on a server equipped with OS 70 (e.g., Host OS), and performs packet forwarding between the NIC 11 of HW 10 connected to OS 70 and the packet processing APL1.

[0011] OS70 includes kernel71, ring buffer72, and driver73, and kernel71 includes protocol processing unit74. Kernel71 is a core function of OS70 (for example, the Host OS), and it monitors hardware and manages the execution status of programs on a process basis. Here, kernel71 responds to requests from packet processing APL1 and also transmits requests from HW10 to packet processing APL1. Kernel71 processes requests from packet processing APL1 via system calls (a user program running in unprivileged mode requests processing from the kernel running in privileged mode). Kernel71 transmits packets to packet processing APL1 via Socket75. Kernel71 receives packets from packet processing APL1 via Socket75.

[0012] Ring Buffer72 is managed by Kernel71 and resides in the server's memory space. Ring Buffer72 is a fixed-size buffer that stores messages output by Kernel71 as a log, and when it exceeds its upper limit, it is overwritten from the beginning.

[0013] Driver73 is a device driver used in kernel71 to monitor hardware. Note that Driver73 depends on kernel71, and if the kernel source code used to create it changes, it will become a different driver. In this case, you will need to obtain the relevant driver source code, rebuild it on the OS that will use the driver, and create the driver from there.

[0014] The protocol processing unit 74 performs protocol processing for L2 (Data Link Layer), L3 (Network Layer), and L4 (Transport Layer) as defined by the OSI (Open Systems Interconnection) reference model.

[0015] Socket75 is the interface that kernel71 uses for inter-process communication. Socket75 has a socket buffer and does not frequently perform data copying operations. The process for establishing communication via Socket75 is as follows: 1. The server creates a socket file to accept clients. 2. The server names the acceptance socket file. 3. The server creates a socket queue. 4. The server accepts the first connection from a client in the socket queue. 5. The client creates a socket file. 6. The client sends a connection request to the server. 7. The server creates a connection socket file in addition to the acceptance socket file. As a result of establishing communication, packet processing APL1 can call system calls such as read() and write() on kernel71.

[0016] In the above configuration, Kernel71 receives notification of packet arrival from NIC11 via a hardware interrupt (hardIRQ) and schedules a software interrupt (softIRQ) for packet processing. The New API (NAPI), implemented in Linux kernel 2.5 / 2.6 and above, processes packets upon arrival using a hardware interrupt (hardIRQ) followed by a software interrupt (softIRQ). As shown in Figure 22, packet forwarding using the interrupt model involves waiting for interrupt processing (see symbol a in Figure 22), which increases the delay in packet forwarding.

[0017] The following describes the overview of packet processing on the NAPI Rx side. [Rx-side packet processing configuration using New API (NAPI)] Figure 23 is a diagram illustrating the overview of Rx-side packet processing by the New API (NAPI) in the area enclosed by the dashed line in Figure 22. <Device driver> As shown in Figure 23, the Device driver includes NIC11 (physical NIC), which is a network interface card; hardIRQ81, which is a handler that is called when a processing request occurs on NIC11 and executes the requested processing (hardware interrupt); and netif_rx82, which is a software interrupt processing function unit.

[0018] <Networking layer> The Networking layer contains softIRQ83, a handler that is called when a processing request occurs in netif_rx82 and executes the requested processing (software interrupt), and do_softirq84, a control function unit that implements the software interrupt (softIRQ). It also contains net_rx_action85, a packet processing function unit that receives and executes software interrupts (softIRQ), poll_list86, which registers net device information indicating which device the hardware interrupt from NIC11 is from, netif_receive_skb87, which creates the sk_buff structure (a structure that allows Kernel71 to perceive the status of packets), and Ring Buffer72.

[0019] <Protocol layer> The protocol layer contains packet processing functions such as ip_rcv88 and arp_rcv89.

[0020] The above netif_rx82, do_softirq84, net_rx_action85, netif_receive_skb87, ip_rcv88, and arp_rcv89 are program components (function names) used for packet processing within Kernel 71.

[0021] [Rx-side packet processing operation using the New API (NAPI)] The arrows (symbols) b to m in Figure 23 indicate the flow of Rx-side packet processing. When the hardware function unit 11a of NIC11 (hereinafter referred to as NIC11) receives a packet (or frame) within a frame from the opposing device, it copies the arriving packet to the Ring Buffer 72 via DMA (Direct Memory Access) transfer without using the CPU (see symbol b in Figure 23). This Ring Buffer 72 is a memory space within the server and is managed by Kernel 71 (see Figure 22).

[0022] However, if NIC11 simply copies the packet that arrives in Ring Buffer72, Kernel71 will not be able to recognize that packet. Therefore, when a packet arrives, NIC11 raises a hardware interrupt (hardIRQ) to hardIRQ81 (see symbol c in Figure 23), and netif_rx82 executes the following process so that Kernel71 can recognize the packet. Note that hardIRQ81, enclosed in an ellipse in Figure 23, represents the handler, not the functional part.

[0023] netif_rx82 is the function that actually performs the processing. When hardIRQ81 (handler) is activated (see symbol d in Figure 23), it saves information about the net device (net_device), which indicates which device the hardware interrupt from NIC11 is from, into poll_list86. Then, netif_rx82 registers queue pruning (referencing the contents of packets accumulated in the buffer and removing the corresponding queue entry from the buffer, taking into account the next processing to be performed on that packet) (see symbol e in Figure 23). Specifically, when netif_rx82 receives that packets have been packed into Ring Buffer72, it uses the NIC11 driver to register subsequent queue pruning in poll_list86. As a result, poll_list86 contains queue pruning information due to packets being packed into Ring Buffer72.

[0024] Thus, in the <Device driver> of FIG. 23, when NIC11 receives a packet, it copies the packet that has arrived at Ring Buffer72 by DMA transfer. Also, NIC11 raises hardIRQ81 (handler), and netif_rx82 registers the net_device in poll_list86 and schedules a software interrupt (softIRQ). So far, the processing of hardware interrupts in the <Device driver> of FIG. 23 stops.

[0025] After that, netif_rx82 raises it to softIRQ83 (handler) with a software interrupt (softIRQ) to harvest the data stored in Ring Buffer72 using the information (specifically, a pointer) in the queue stored in poll_list86 (see reference f in FIG. 23), and notifies do_softirq84, which is the control function part of the software interrupt (see reference g in FIG. 23).

[0026] do_softirq84 is a software interrupt control function part that defines each function of the software interrupt (there are various packet processes, and interrupt processing is one of them. Define interrupt processing). Based on this definition, do_softirq84 notifies net_rx_action85, which actually performs software interrupt processing, of the request for the current (corresponding) software interrupt (see reference h in FIG. 23).

[0027] When the order of softIRQs comes around, net_rx_action85 calls a polling routine to harvest packets from Ring Buffer72 based on the net_device registered in poll_list86 (see reference i in FIG. 23) and harvests the packets (see reference j in FIG. 23). At this time, net_rx_action85 continues harvesting until poll_list86 becomes empty. After that, net_rx_action85 notifies netif_receive_skb87 (see reference k in FIG. 23).

[0028] netif_receive_skb87 creates an sk_buff structure, analyzes the contents of the packet, and passes the processing to the subsequent protocol processing unit 74 (see Figure 22) according to the type. In other words, netif_receive_skb87 analyzes the contents of the packet and, when processing according to the contents of the packet,<Protocol layer> The process is then passed to ip_rcv88 (symbol l in Figure 23), and if it is L2, for example, the process is passed to arp_rcv89 (symbol m in Figure 23).

[0029] Patent Document 1 describes a server-internal network delay control device (KBP: Kernel Busy Poll). KBP constantly monitors packet arrivals within the kernel using a polling model. This suppresses softIRQ and enables low-latency packet processing. [Prior art documents] [Patent Documents]

[0030] [Patent Document 1] International Publication No. 2021 / 130828 [Non-patent literature]

[0031] [Non-Patent Document 1] New API (NAPI), [online], [searched April 4, 2022], Internet < URL: http: / / http: / / lwn.net / 2002 / 0321 / a / napi-howto.php3 > [Overview of the project] [Problems that the invention aims to solve]

[0032] However, both the interrupt model and the polling model for packet forwarding have the following challenges. The interrupt model uses software interrupt processing to process packets, which is performed by the kernel upon receiving an event (hardware interrupt) from the hardware. Therefore, because the interrupt model relies on interrupt (software interrupt) processing for packet forwarding, it faces challenges such as conflicts with other interrupts and delays if the target CPU is occupied by a higher-priority process. In this case, congestion in interrupt processing further increases the waiting delay. For example, packet forwarding using an interrupt model involves waiting for interrupt processing to perform packet forwarding, which increases the delay in packet forwarding.

[0033] This section provides additional information about the mechanism by which delays occur in interrupt models. In a typical kernel, packet forwarding is handled by a hardware interrupt followed by a software interrupt. When a software interrupt occurs during packet forwarding, the software interrupt process cannot be executed immediately under the following conditions (1) to (3). Therefore, the interrupt process is arbitrated by a scheduler such as ksoftirqd (a kernel thread for each CPU that is executed when the software interrupt load becomes high), and the interrupt process is scheduled, resulting in a wait of the order of milliseconds. (1) When it conflicts with other hardware interrupt handlers (2) When it conflicts with other software interrupt handlers (3) When other processes with higher priority, kernel threads (migration threads, etc.), or the interrupt target CPU are in use. Under the above conditions, the software interrupt processing cannot be executed immediately.

[0034] Similarly, packet processing using the New API (NAPI) also results in network delays on the order of milliseconds due to contention in interrupt handling (softIRQ), as shown in the dashed box n in Figure 23. On the other hand, using the technology described in Patent Document 1, software interrupts can be suppressed and low-latency packet harvesting can be achieved by constantly monitoring packet arrival. However, since monitoring packet arrival occupies a CPU core and uses CPU time, power consumption is high. In other words, there is a problem that power consumption is high because the kernel thread that constantly monitors packet arrival occupies a CPU core and always uses CPU time. The relationship between workload and CPU utilization will be explained with reference to Figures 24 and 25.

[0035] Figure 24 shows an example of video (30 FPS) data transfer. The workload shown in Figure 24 has a transfer rate of 350 Mbps and performs intermittent data transfer every 30 ms.

[0036] Figure 25 shows the CPU usage rate used by the busy poll thread in the KBP described in Patent Document 1. As shown in Figure 25, in KBP, the kernel thread occupies a CPU core to perform busy polling. Even with intermittent packet reception as shown in Figure 24, KBP always uses the CPU regardless of whether packets have arrived or not, which leads to the problem of high power consumption.

[0037] In light of this background, the present invention was made, and its objective is to reduce power consumption by suppressing the excessive generation of hardware interrupts by putting the polling thread to sleep, while also reducing latency within the server and enabling packet forwarding. [Means for solving the problem]

[0038] To solve the aforementioned problems, a server-internal delay control device is provided that is located in the kernel space of the OS and processes data by launching a thread that monitors data arrival using a polling model, comprising: a data arrival monitoring unit that monitors a poll list that registers net device information indicating which device a hardware interrupt from the interface unit is from; a data harvesting unit that, when data has arrived, refers to the data held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer; a sleep management unit that puts the thread to sleep and wakes the thread from sleep by a hardware interrupt when data arrives; and a management unit that, when the timing of data arrival can be predicted, performs pre-wake control to wake the thread immediately before the data arrives in accordance with the timing of data arrival. [Effects of the Invention]

[0039] According to the present invention, excessive HW interrupts are suppressed by putting the polling thread to sleep, reducing power consumption and enabling packet forwarding with reduced latency within the server. [Brief explanation of the drawing]

[0040] [Figure 1] This is a schematic diagram of the server-internal delay control system according to the first embodiment of the present invention. [Figure 2] Figure 1 shows an example configuration where the polling thread (server-internal delay control device) is placed in kernel space. [Figure 3] Figure 1 shows an example configuration where the polling thread (server-internal delay control device) is placed in User Space. [Figure 4] This figure shows an example of the polling thread operation of the in-server delay control device of the in-server delay control system according to the first embodiment of the present invention. [Figure 5]This figure shows an example of the C-state state of a server-internal delay control system according to the first embodiment of the present invention. [Figure 6] This figure shows the traffic image and the C-state transition image of the polling thread when the packet arrival interval of the server-internal delay control system according to the first embodiment of the present invention is dense. [Figure 7] This figure shows a traffic image and a C-state transition image of the polling thread when the packet arrival interval of the server-internal delay control system according to the first embodiment of the present invention is sparse. [Figure 8] A server-internal delay control system according to the first embodiment of the present invention <periodically wakes up the polling thread> (<method <1> This figure shows the traffic image, the C-state transition image of the polling thread, and the polling thread's wake-up in the case of ). [Figure 9] <Wake up the polling thread in advance> (method) of the server-internal delay control system according to the first embodiment of the present invention <2> This figure shows the traffic image, the C-state transition image of the polling thread, and the polling thread's wake-up in the case of ). [Figure 10] This is a flowchart showing the method selection process of the management unit of the idle state return control unit of the server-internal delay control device of the server-internal delay control system according to the first embodiment of the present invention. [Figure 11] This flowchart shows the NIC and HW interrupt processing of the server-internal delay control device of the server-internal delay control system according to the first embodiment of the present invention. [Figure 12] This flowchart shows the processing of the wake-up logic operation mode of the server-internal delay control device of the server-internal delay control system according to the first embodiment of the present invention, taking into account CPU idle. [Figure 13] This is a schematic diagram of a server-internal delay control system according to a second embodiment of the present invention. [Figure 14]This figure illustrates an example of time allocation for DownLink and UpLink in TDD of a server-internal delay control system according to the second embodiment of the present invention. [Figure 15] This figure shows a table of subcarrier intervals and symbol intervals for each Numerology in the server-internal delay control system according to the second embodiment of the present invention. [Figure 16] This flowchart shows the processing of the operation mode of the wake-up logic of the server-internal delay control device of the server-internal delay control system according to the second embodiment of the present invention, taking into account CPU idleness. [Figure 17] This hardware configuration diagram shows an example of a computer that implements the functions of an in-server delay control device in an in-server delay control system according to an embodiment of the present invention. [Figure 18] This figure shows an example of applying an in-server delay control system, which places a polling thread within the kernel, to the interrupt model in a server virtualization environment using a general-purpose Linux kernel (registered trademark) and VM configuration. [Figure 19] This figure shows an example of applying an in-server delay control system, which places a polling thread within the kernel, to the interrupt model in a containerized server virtualization environment. [Figure 20] This figure shows an example of applying an in-server delay control system that places polling threads in user space to an interrupt model in a server virtualization environment using a general-purpose Linux kernel (registered trademark) and VM configuration. [Figure 21] This figure shows an example of applying an in-server delay control system that places polling threads in user space to an interrupt model in a containerized server virtualization environment. [Figure 22] This is a schematic diagram of Rx-side packet processing using the New API (NAPI) implemented in Linux kernel 2.5 / 2.6. [Figure 23] Figure 22 shows an overview of Rx-side packet processing by the New API (NAPI) in the area enclosed by the dashed line. [Figure 24] This figure shows an example of video (30 FPS) data transfer. [Figure 25] This figure shows the CPU usage rate used by the busy poll thread in the KBP described in Patent Document 1. [Modes for carrying out the invention]

[0041] The following describes a server-internal delay control system, etc., in an embodiment for carrying out the present invention (hereinafter referred to as "this embodiment") with reference to the drawings. [overview] (First Embodiment) Figure 1 is a schematic diagram of a server-internal delay control system according to the first embodiment of the present invention. This embodiment is an example applied to Rx-side packet processing using the New API (NAPI) implemented in Linux kernel 2.5 / 2.6. Components identical to those in Figure 22 are denoted by the same reference numerals. As shown in Figure 1, the server-internal delay control system 1000 executes a packet processing APL1 located in the user space available to the user on a server equipped with an OS (e.g., Host OS), and performs packet forwarding between the NIC 11 of the hardware connected to the OS and the packet processing APL1.

[0042] The server-internal delay control system 1000 comprises a network interface card NIC 11 (physical NIC), a handler hardIRQ 81 which is called when a processing request occurs on the NIC 11 and executes the requested processing (hardware interrupt), a hardware interrupt processing unit 182 which is a hardware interrupt processing function unit, a receive list 186, a Ring_Buffer 72, a polling thread (server-internal delay control device 100), and a protocol processing unit 74. Ring Buffer72 is managed by the kernel in the server's memory space. It is a fixed-size buffer that stores messages output by the kernel as a log, and when it exceeds its upper limit, it is overwritten from the beginning. The protocol processing unit 74 handles Ethernet, IP, TCP / UDP, etc. The protocol processing unit 74 performs L2 / L3 / L4 protocol processing as defined by, for example, the OSI reference model.

[0043] <Server-based delay control device> The server-internal delay control device 100 is a polling thread located in either kernel space or user space. The server-internal delay control device 100 includes a packet arrival monitoring unit 110, a packet harvesting unit 120, a sleep management unit 130, a CPU frequency / CPU idle setting unit 140, an idle state return control unit 150, and a management unit 160.

[0044] The packet arrival monitoring unit 110 is a thread that monitors whether packets have arrived. The packet arrival monitoring unit 110 monitors (polls) the receive list 186.

[0045] The packet arrival monitoring unit 110 obtains pointer information and net_device information indicating the presence of a packet in Ring_Buffer 72 from the receive list 186, and transmits this information (pointer information and net_device information) to the packet harvesting unit 120. If multiple packet information exists in the receive list 186, the unit transmits the information for each packet.

[0046] If a packet has arrived, the packet harvesting unit 120 refers to the packet held in the Ring Buffer 72 and performs harvesting by deleting the corresponding queue entry from the Ring Buffer 72 based on the next processing to be performed (hereinafter, this may simply be referred to as harvesting packets from the Ring Buffer 72). Based on the received information, the packet harvesting unit 120 retrieves the packet from the Ring Buffer 72 and transmits the packet to the protocol processing unit 74. When multiple packets are stored in Ring_Buffer 72, the packet harvesting unit 120 harvests multiple packets at once and passes them to the subsequent protocol processing unit 74. The number of packets harvested at once is called a quota, and this process is often referred to as batch processing. The protocol processing unit 74 also processes multiple packets at once, making it fast.

[0047] The sleep management unit 130 puts a polling thread to sleep if no packets arrive for a predetermined period of time, and wakes the polling thread from sleep via a hardware interrupt (hardIRQ) when a packet arrives (details below).

[0048] The CPU frequency / CPU idle setting unit 140 lowers the CPU operating frequency of the CPU core used by the polling thread while the system is asleep. The CPU frequency / CPU idle setting unit 140 also sets the CPU idle state of the CPU core used by this polling thread to power-saving mode while the system is asleep (details below).

[0049] The idle state recovery control unit 150 wakes up the thread periodically during sleep mode, or wakes up the thread immediately before a packet arrives, in accordance with the timing of packet arrival (details below). An example of the idle state recovery control unit 150 periodically waking up the thread during sleep mode will be described in this embodiment, and an example of the idle state recovery control unit 150 waking up the thread immediately before a packet arrives, in accordance with the timing of packet arrival, will be described in the second embodiment.

[0050] The management unit 160 selects either a periodic wake-up control, which wakes the thread periodically during sleep mode, or a pre-wake-up control, which wakes the thread immediately before a packet arrives, in accordance with the packet arrival timing (details below). Alternatively, the management unit 160 may choose not to select either the periodic wake-up control or the pre-wake-up control, and instead select the conventional method.

[0051] <Placement of delay control devices within the server> Figures 2 and 3 illustrate the arrangement of the polling thread (server-internal delay control device 100) in Figure 1. • Kernel space allocation for polling threads Figure 2 shows an example configuration in which the polling thread (server-internal delay control device 100) from Figure 1 is placed in kernel space. The server-internal delay control system 1000 shown in Figure 2 has a polling thread (server-internal delay control device 100) and a protocol processing unit 74 located in kernel space. This polling thread (server-internal delay control device 100) operates within kernel space. The server-internal delay control system 1000 executes a packet processing APL1 located in user space on a server equipped with an OS, and performs packet forwarding between the HW's NIC 11 and the packet processing APL1 via a device driver connected to the OS. As shown in Figure 2, the Device driver includes hardIRQ81, HW interrupt processing unit 182, receive list 186, and Ring_Buffer 72. A device driver is a driver used to monitor hardware.

[0052] The idle state recovery control unit 150 of the server's delay control device 100, in a configuration where a polling thread is located within the kernel, periodically wakes up the thread during sleep, or wakes up the thread immediately before the arrival of a packet in accordance with the packet arrival timing. The idle state recovery control unit 150 manages hardware interrupts and controls the sleep state of the polling thread and enables / denies hardware interrupts via hardIRQ81 (see the symbol xx in Figure 2).

[0053] This invention can be applied to systems like NAPI and KBP, where there is a polling thread within the kernel.

[0054] • Userspace placement of polling threads Figure 3 shows an example configuration in which the polling thread (server-internal delay control device 100) from Figure 1 is placed in User Space. The server-internal delay control system 1000 shown in Figure 3 has a polling thread (server-internal delay control device 100) and a protocol processing unit 74 located in user space. This polling thread (server-internal delay control device 100) operates in user space, not in kernel space. The server-internal delay control system 1000 shown in Figure 3 uses a polling thread (server-internal delay control device 100) to bypass the kernel space and perform packet forwarding between the device driver, NIC 11, and packet processing APL1.

[0055] The idle state recovery control unit 150 of the server's delay control device 100, in a configuration where a polling thread is placed in user space, periodically wakes up the thread during sleep, or wakes up the thread immediately before the arrival of a packet in accordance with the packet arrival timing. The idle state recovery control unit 150 manages hardware interrupts and controls the sleep state of the polling thread and enables / denies hardware interrupts to the hardware interrupt processing unit 182 (see the symbol yy in Figure 3).

[0056] The present invention can be applied when there is a polling thread in user space, such as in DPDK.

[0057] The operation of the server-internal delay control system 1000, configured as described above, will be explained below. This invention can be applied to both cases where the polling thread is located within the kernel, as in NAPI and KBP, and cases where the polling thread is located in user space, as in DPDK. We will explain its application to cases where the polling thread is located within the kernel as an example.

[0058] [Rx-side packet processing operation according to the present invention] The arrows (symbols) aa to ii in Figures 1 to 3 indicate the flow of Rx-side packet processing. When NIC11 receives a packet (or frame) within a frame from the peer device, it copies the arriving packet to Ring Buffer72 via DMA transfer without using the CPU (see symbol aa in Figures 1-3). This Ring Buffer72 is<Device driver> It is managed by [this method].

[0059] When a packet arrives, NIC11 raises a hardware interrupt (hardIRQ) to hardIRQ81 (handler) (see code bb in Figures 1 to 3), and the HW interrupt processing unit 182 recognizes the packet by executing the following process.

[0060] When the hardwire81 (handler) is activated (see code cc in Figure 1), the HW interrupt processing unit 182 saves the net device information, which indicates which device the hardware interrupt from NIC11 is from—one of the pieces of information in the hardware interrupt (hardIRQ)—to the receive list 186, and registers the queue plucking information. Specifically, when the HW interrupt processing unit 182 receives that packets have been packed into the Ring Buffer 72, it uses the NIC11 driver to register subsequent queue plucking in the receive list 186 (see code dd in Figures 1 to 3). As a result, the queue plucking caused by packets being packed into the Ring Buffer 72 is registered in the receive list 186.

[0061] The hardware interrupt processing unit 182 registers net_device in the receive list 186, but unlike netif_rx82 in Figure 23, it does not schedule software interrupts (softIRQ). In other words, the hardware interrupt processing unit 182 differs from netif_rx82 in Figure 23 in that it does not schedule software interrupts (softIRQ).

[0062] Furthermore, the HW interrupt processing unit 182 wakes up the sleeping polling thread (see code ee in Figures 1 to 3). Up to this point, Figures 1 to 3<Device driver> Hardware interrupt processing in this case will be stopped.

[0063] In this embodiment, as shown in Figure 23<Networking layer> In this configuration, softIRQ83 and do_softirq84 are removed, and consequently, netif_rx82, as shown in Figure 23, no longer sends a notification to start softIRQ83 (handler) (see symbol f in Figure 23).

[0064] In this embodiment, the server-internal delay control system 1000 removes softIRQ83 and do_softirq84 shown in Figure 23, and instead<kernel space> A polling thread (server-internal delay control device 100) is provided (see Figure 2). Alternatively, the server-internal delay control system 1000 is:<User space> A polling thread (server-internal delay control device 100) is provided (see Figure 3).

[0065] The packet arrival monitoring unit 110 monitors (polling) the receive list 186 (see the symbol ff in Figures 1 to 3) to confirm whether or not a packet has arrived. The packet arrival monitoring unit 110 obtains pointer information and net_device information indicating the presence of a packet in Ring_Buffer 72 from the receive list 186 and transmits this information (pointer information and net_device information) to the packet harvesting unit 120 (see the symbol gg in Figures 1 to 3). If there is information for multiple packets in the receive list 186, the unit transmits the information for each packet.

[0066] The packet harvesting unit 120 of the server's delay control device 100 harvests packets from the Ring Buffer 72 if a packet has arrived (see the symbol hh in Figures 1 to 3). The packet harvesting unit 120 extracts packets from the Ring_Buffer 72 based on the received information and transmits the packets to the protocol processing unit 74 (see code ii in Figures 1 to 3).

[0067] [Sleep behavior of polling threads] The server-internal delay control system 1000 stops the softIRQ for packet processing, which is the main cause of network delays, and the packet arrival monitoring unit 110 of the server-internal delay control device 100 executes a polling thread to monitor packet arrivals. Then, the packet harvesting unit 120 processes packets using the polling model (without softIRQ) when they arrive.

[0068] When a packet arrives, a polling thread is triggered by a hardware interrupt handler, avoiding softIRQ contention and enabling immediate packet forwarding. In other words, by keeping the packet arrival monitoring function on standby and triggering it with a hardware interrupt, it is possible to achieve lower latency than packet forwarding processing using software interrupts such as NAPI.

[0069] The polling thread (server-internal delay control device 100) that monitors packet arrivals can sleep when no packets are arriving. The polling thread (server-internal delay control device 100) sleeps depending on whether a packet has arrived, and wakes up when a packet arrives using hardIRQ81. Specifically, the sleep management unit 130 of the server-internal delay control device 100 puts the polling thread to sleep depending on whether a packet has arrived, that is, if no packets have arrived for a predetermined period of time. The sleep management unit 130 wakes up when a packet arrives using hardIRQ81. This avoids softIRQ contention and achieves low latency.

[0070] The CPU frequency / CPU idle setting unit 140 of the server's internal delay control device 100 changes the CPU operating frequency and idle settings depending on whether a packet has arrived or not. Specifically, the CPU frequency / CPU idle setting unit 140 lowers the CPU frequency when the server is asleep and increases the CPU frequency (returns the CPU operating frequency to its original value) when the server is started again. In addition, the CPU frequency / CPU idle setting unit 140 changes the CPU idle setting to power saving when the server is asleep. Power saving is achieved by lowering the CPU operating frequency and changing the CPU idle setting to power saving when the server is asleep.

[0071] Figure 4 shows an example of the operation of the polling thread of the server's delay control device 100. The vertical axis shows the CPU utilization rate [%] of the CPU core used by the polling thread, and the horizontal axis shows time. Figure 4 shows an example of the polling thread operation due to packet arrival, corresponding to the data transfer example of video (30 FPS) where packets are received intermittently, as shown in Figure 24. As shown in Figure 4, the sleep management unit 130 of the server's delay control device 100 puts the polling thread to sleep if no packets arrive for a predetermined period of time (more specifically, if no further packets arrive after a fixed period predetermined by the maintenance / operator has elapsed since the arrival of a packet) (see symbol p in Figure 4). The sleep management unit 130 then activates the polling thread upon the arrival of a packet using hardIRQ81 (see symbol q in Figure 4).

[0072] During sleep mode, the kernel thread does not exclusively use the CPU core. Therefore, in addition to being used by the polling thread, the CPU usage of the CPU core may fluctuate due to timer interrupts for system stability or migration threads for error handling (see symbol r in Figure 4).

[0073] [LPI (Low Power Idle) Hardware Control] The CPU has a function to control the CPU's idle state through hardware control, which is called LPI. LPI is often also called CPUidle or C-state, and below, LPI will be explained as C-state. C-state attempts to conserve power by turning off the power to some of the CPU's circuits when the CPU load is low.

[0074] Figure 5 is a table illustrating an example of a C-state. Note that the state definition varies depending on the CPU hardware, so Figure 5 is for reference only. As shown in Figure 5, the CPUidle state has grades C0 to C6, and as the time without CPU load increases, it transitions to a deeper sleep state. While deeper sleep states consume less CPU power, they also increase the time required to recover, which can pose a challenge from a low-latency perspective. Furthermore, the depth to which the CPUidle state transitions is controlled by the CPU hardware and is dependent on the CPU product (often not controllable by software such as the kernel).

[0075] [Relationship between sleep control and C-state state] When a polling thread is put into sleep mode, the CPU transitions to the idle state (each state of the C-state) via LPI hardware control during periods of no traffic inflow. The longer the period without traffic, the deeper the idle state becomes. In this case, the logic is CPU hardware dependent (CPU product dependent) and cannot be controlled by software.

[0076] Figure 6 shows the traffic image (top image) and the C-state transition image of the polling thread when the packet arrival interval is close (bottom image). As shown in the upper part of Figure 6, when the packet arrival interval is close, the transition to the idle state (each state of C-state) is limited to C2 of C-state (see the symbol s in the lower part of Figure 6).

[0077] Figure 7 shows the traffic image (top of Figure 7) and the C-state transition image of the polling thread when the packet arrival interval is sparse (bottom of Figure 7). When the packet arrival interval is sparse, as shown in the upper diagram of Figure 7, the transition to the idle state (each state of C-state) drops to a deeper idle state (here, the highest grade C6) due to the longer period without traffic (see code t in the lower diagram of Figure 7) (see code u in the lower diagram of Figure 7).

[0078] If the packet arrival interval is sparse, the system will fall into a deep idle state, and it will take time to recover (wake up from deep sleep) (see arrow v in the lower part of Figure 7). Thus, in the case of sparse packet traffic, the system can fall into a deep idle state, and recovering from this state takes time, which can lead to delays in packet forwarding processing. This is a problem.

[0079] [Basic concept of the operation of the polling thread (server-internal delay control device 100)] Referring to Figures 8 and 9, the basic concept of the operation of the polling thread (server-internal delay control device 100) will be explained. This invention prevents the CPU from entering a deep idle state due to prolonged periods without packet arrival by periodically waking up the polling thread while the CPU is in sleep mode. <1> (Figure 8), or wake up the polling thread in advance (method) <2> (Figure 9). Note that this method involves waking up the polling thread in advance. <2> In this case, the timing of waking up can be determined using, for example, a timer. A method that periodically wakes up the polling thread. <1> This will be explained in this embodiment, which describes a method of waking up the polling thread in advance. <2> This will be described later in the second embodiment.

[0080] <Periodically wake up the polling thread> (<Method <1> ) Figure 8 shows the method of <periodically waking up the polling thread>. <1> This figure shows the traffic image (top image in Figure 8) when the polling thread is activated, the C-state transition image of the polling thread, and the polling thread waking up (bottom image in Figure 8). In the case of sparse packet arrival traffic as shown in the upper part of Figure 8, the polling thread is periodically woken up ("periodic wake-up") as indicated by the symbol jj in the lower part of Figure 8. Periodic wake-up prevents the system from falling into a deep idle state. In conventional cases, when the packet arrival interval is sparse, the system falls into a deep idle state (see arrow t in the lower part of Figure 7), and it takes time to recover (see arrow v in the lower part of Figure 7). In contrast, the method <1> Therefore, when the packet arrival interval is sparse, the polling thread is periodically woken up ("periodic wake-up") to prevent the CPU from entering a deep idle state. This prevents the CPU from falling into a deep idle state, and makes it possible to speed up the recovery time from the idle state.

[0081] <Wake up the polling thread in advance> (method) <2> ) Figure 9 shows the method of <waking up the polling thread in advance>. <2> This figure shows the traffic image (top of Figure 9) when the polling thread is activated, the C-state transition image of the polling thread, and the polling thread waking up (bottom of Figure 9). In the case of sparse packet traffic as shown in the upper part of Figure 9, the polling thread is woken up in advance by a timer, for example, as indicated by the code kk in the lower part of Figure 9 ("pre-wake"). Pre-wake aims to achieve power saving as much as possible by dropping the system into a deep idle state, and then wakes up just before packet arrival to restore the C-state, thereby speeding up the recovery time from the idle state.

[0082] <“Regular wake-up” (method <1> ) and "pre-wake-up" (method) <2> ) contrast > “Regular wake-up” (method <1> This is suitable for applications where the timing of packet arrival cannot be predicted. “Wake up in advance” (method <2> This is suitable for applications where packet arrival timing can be predicted. One example of such an application is a RAN (Radio Access Network). Here, by limiting the process to simply waking up during scheduled / pre-arrival wake-ups, it is possible to avoid unnecessary CPU cycle consumption and minimize the increase in power consumption due to waking up.

[0083] [Operation flow of the polling thread (server-internal delay control device 100)] <Method Selection Process> Figure 10 is a flowchart showing the method selection process of the management unit 160 of the idle state return control unit 150 of the polling thread (server-internal delay control device 100). The polling thread checks for packet arrival or initiates this flow periodically.

[0084] In step S1, the management unit 160 of the idle state return control unit 150 determines the operating mode of the wake-up logic considering CPU idle (conventional method / method). <1> / method <2> ) is selected and the information necessary to switch the operating mode is collected. The management unit 160 collects, for example, data measured by the traffic measurement unit 220 (Figure 13), traffic patterns known in advance as part of the service specifications, planned event information related to user behavior, etc. (conventional method / method) <1> / method <2> Select ). In step S2, the management unit 160 determines whether low latency is required for the next arriving traffic.

[0085] If low latency is not required for the next arriving traffic (S2: No), in step S3, the management unit 160 determines that there is no need for control that takes low latency into consideration, selects the conventional method, and terminates the processing of this flow.

[0086] If low latency is required for the next arriving traffic (S2: Yes), in step S4 the management unit 160 determines whether the future traffic is predictable or not.

[0087] If future traffic is unpredictable (S4: No), in step S5 the control unit 160 will implement the following method <1> Select this option to terminate the processing of this flow.

[0088] If future traffic can be predicted (S4: No), in step S6, the management unit 160 selects Method <2> and ends the processing of this flow.

[0089] <NIC and HW Interrupt Processing> FIG. 11 is a flowchart showing the NIC and HW interrupt processing of a polling thread (server internal delay control device). While the polling thread is running, this operation flow is looped and executed. When a packet arrives at the NIC 11, this flow starts. In step S11, the NIC 11 copies the packet data that has arrived by DMA (Direct Memory Access) to a memory area.

[0090] In step S12, the polling thread (server internal delay control device 100) determines whether the HW interrupt is permitted. If the HW interrupt is permitted (S12: Yes), it proceeds to step S13. If the HW interrupt is not permitted (S12: No), the processing of this flow ends. In step S13, the NIC 11 raises a HW interrupt (hardIRQ) to the hardIRQ81 (handler) to activate the HW interrupt, and registers packet arrival information (such as NIC device information) in the receive list 186. In step S14, if the polling thread (server internal delay control device 100) is sleeping, the NIC 11 wakes up the polling thread and ends the processing of this flow.

[0091] <Operation Flow of Polling Thread> FIG. 12 is a flowchart showing the processing of the operation mode of the wake-up logic considering the CPU idle of a polling thread (server internal delay control device). When the polling thread is sleeping, a packet arrives and is caused by a HW interrupt (HW interrupt at point 1), and this flow starts. In step S21, the idle state recovery control unit 150 disables hardware interrupts by the NIC 11. If a hardware interrupt occurs during processing, the processing will be interrupted, so the idle state recovery control unit 150 temporarily disables hardware interrupts by the NIC 11.

[0092] In step S22, the CPU frequency / CPU idle setting unit 140 sets the CPU frequency of the CPU core on which the polling thread operates to a higher value, and if the corresponding CPU was in an idle state, it releases the idle state.

[0093] In step S23, the polling thread refers to receive list 186. The polling thread learns which device the hardware interrupt originated from and checks the packet arrival information in receive list 186 in the next step, S24. Alternatively, instead of referring to the Control Plane list called receive list186, you can directly refer to Ring Buffer72 to check for packet arrivals. For example, NAPI, implemented in the Linux kernel, monitors the Control Plane list called poll_list.

[0094] In step S24, the packet arrival monitoring unit 110 determines whether or not packet arrival information exists in the receive list 186. If there is no packet arrival information in the receive list 186 (S24: No), that is, if there are no packets to process, the following processing is skipped and the process proceeds to step S27.

[0095] If simply waking up the polling thread does not cause the C-state to transition to C0 or similar, you may perform a light calculation here to load the CPU. Additionally, executing the flow triggered by the hardware interrupt will also load the CPU. Therefore, you may execute the same flow as when triggered by the hardware interrupt at point 1.

[0096] If packet arrival information exists in receive list 186 (S24: Yes), in step S25 the polling thread references the packet data from ring buffer 72 and forwards the corresponding data to the subsequent protocol processing unit 74. If there are multiple data, they may be received and processed all at once.

[0097] In step S26, the packet harvesting unit 120 determines whether or not there are unreceived packets in the ring buffer 72. If there are unreceived packets in the ring buffer 72 (S26: Yes), the process returns to step S25.

[0098] If there are no unreceived packets in the ring buffer 72 (S26: No), in step S27, the CPU frequency / CPU idle setting unit 140 sets the CPU frequency of the CPU core on which the polling thread operates to a lower value and puts the corresponding CPU into an idle state.

[0099] In step S28, the packet arrival monitoring unit 110 removes the corresponding NIC information from the receive list 186.

[0100] In step S29, the idle state recovery control unit 150 allows the corresponding NIC to perform a hardware interrupt.

[0101] In step S30, the idle state recovery control unit 150 sets a timer t to periodically wake up the polling thread so that the CPU idle state does not become too deep.

[0102] The above value t may be set in advance by the operator according to specifications such as the time required for C-state transitions controlled by the CPU hardware, and the time required to return from each idle state. Furthermore, if these change dynamically, the state information of the C-state can be obtained from the CPU, the C-state transition time can be learned from traffic conditions, etc., and t can be dynamically determined from this learned time.

[0103] In step S31, the sleep management unit 130 puts the polling thread to sleep. In step S32, the idle state return control unit 150 determines whether or not timer t has expired. If timer t has not expired (S32: No), it returns to step S31. If timer t has expired (S32: Yes), it proceeds to step S33.

[0104] In step S33, the idle state return control unit 150 wakes up the polling thread when timer t expires and terminates the processing of this flow. When timer t expires, the polling thread wakes up (it simply wakes up, and the C-state returns to C0, etc.). Subsequently, the timer repeatedly wakes up and sleeps. If a packet arrives during this process, a hardware interrupt is triggered, and the system transitions to (HW interrupt at point 1: the start of the flow in Figure 10).

[0105] (Second Embodiment) A second embodiment of the present invention is the "pre-wake-up" (method) described above. <2> This is an example of applying the ). Figure 13 is a schematic diagram of a server-internal delay control system according to a second embodiment of the present invention. The same reference numerals are used for components identical to those in Figure 1. As shown in Figure 13, the server-internal delay control system 1000A includes a server-internal delay control device 200. The server-internal delay control device 200 is a polling thread located in either kernel space or user space. The server-internal delay control device 200 further includes a packet arrival monitoring unit 210 and a traffic measurement unit 220 in addition to the server-internal delay control device 100 shown in Figure 1.

[0106] The packet arrival monitoring unit 210 has the same functions as the packet arrival monitoring unit 110 in Figure 1. The traffic measurement unit 220 measures the traffic of incoming and outgoing packets, such as incoming packets.

[0107] [Specific example of a timer design] As mentioned above, "pre-wake-up" (method) <2> This is suitable for applications where packet arrival timing can be predicted. An example of such a case is a RAN (Range of Random Area). If the packet arrival timing can be predicted, a timer t can be set. The following example uses a vRAN use case. Specifically, the method <2> This document presents a design example for timer t in the UpLink interface of a RAN FrontHaul, which is a target use case.

[0108] Figure 14 illustrates an example of DownLink and UpLink time allocation in TDD (Time Division Duplex). Figure 14 is a subframe 2, 7 (non-patent literature:<https: / / www.sharetechnote.com / html / 5G / 5G_FrameStructure.html> This is an example where an UpLink subframe is assigned to (see reference). In Figure 14, D represents DownLink, S represents Special Subframe, and U represents UpLink. Also, ● indicates sleep, and ▲ indicates wake up for C-state.

[0109] Depending on the RAN multiplexing scheme, in the case of time-division multiplexing, time slots used for DownLink and UpLink are fixedly allocated. Therefore, since no UpLink data arrives except for subframes allocated to UpLink, the system can sleep during this period by setting the time for timer t1. During the period of subframes allocated to UpLink, the system can sleep by setting the time for timer t2.

[0110] • Timer t1 In the time allocation shown in Figure 14 (upper diagram), the time elapsed from the ● mark (sleep) to the ▲ mark (wake up for C-state) is the time elapsed for timer t1. Timer t1 = 4 × t_sub - t_recover For example, if t_sub=1ms and t_recover=30us, then t1 is approximately 3.97ms. however, Subframe interval = t_sub Expected time to recover from C-state = t_recover Symbol interval = t_symbol (Specific values ​​are available in the Numerology table in Figure 15)

[0111] Figure 15 is a table showing the subcarrier spacing and symbol spacing for each Numerology. For example, Numerology 0 has a subcarrier spacing of 15 kHz and a 1 symbol spacing of 71.4 usec. As shown in Numerology 1, 2, 3, ..., the 1 symbol spacing [usec] is halved as the subcarrier spacing [kHz] doubles.

[0112] • Timer T2 Timer t2 = t_symbol - t_recover If Numerology=1 and t_recover=5us, then t2 is approximately 30us.

[0113] [Arrival timing of vRAN traffic] method <2> The target use case for vRAN manages radio resources in the frequency and time domains, with frames arriving at symbol time intervals, which are the smallest units in the time domain. Therefore, it is expected that polling threads will be woken up in advance and restored from the C-state according to the symbol interval. However, since the arrival of frames changes depending on whether the terminal is assigned or not, frames do not necessarily arrive at symbol intervals. Therefore, waking up every symbol interval includes the meaning of speculative waking up. However, since the number of CPU cycles associated with waking up is small, speculative waking up does not significantly contribute to an increase in power consumption.

[0114] Hereinafter, the operation of the in-server delay control system 1000A configured as described above will be described. The overall operation of the in-server delay control system 1000A is the same as that of the in-server delay control system 1000 in FIG. 1. Also, since the <NIC and HW interrupt processing> is the same as that in FIG. 11, the description thereof will be omitted.

[0115] <Operation flow of polling thread> FIG. 16 is a flowchart showing the processing of the operation mode of the wake-up logic considering the CPU idle of the polling thread (in-server delay control device). Steps that perform the same processing as the operation flow in FIG. 12 are denoted by the same reference numerals, and the description of overlapping parts will be omitted. When the polling thread is sleeping, a packet arrives and is caused by an HW interrupt (HW interrupt at point 1), and this flow starts.

[0116] In step S25, the polling thread refers to the packet data from the ring buffer 72, transfers the corresponding data to the subsequent protocol processing unit 74, and proceeds to step S41. In step S41, the traffic measurement unit 220 (FIG. 13) of the packet arrival monitoring unit 210 measures the traffic volume.

[0117] In step S42, the idle state return control unit 150 (FIG. 13) calculates the timer t based on the measured traffic volume.

[0118] In step S43, the packet harvesting unit 120 determines whether or not there are unreceived packets in the ring buffer 72. If there are unreceived packets in the ring buffer 72 (S43: Yes), the process returns to step S25.

[0119] In step S29, the idle state recovery control unit 150 allows a hardware interrupt by the corresponding NIC and proceeds to step S44.

[0120] In step S44, the idle state recovery control unit 150 sets a timer t calculated based on the traffic amount and wakes up the polling thread to prevent the CPU idle state from becoming too deep.

[0121] As described above, the above value t is set so that the system wakes up just before the next packet arrives, depending on the traffic conditions of the service to which the present invention is applied and the time required for each idle state to recover. In addition, if the packet arrival is irregular, the packet arrival timing may be learned and t may be determined using the results of this learning.

[0122] In step S33, the idle state return control unit 150 wakes up the polling thread when timer t expires and terminates the processing of this flow. When timer t expires, the polling thread wakes up (simply by waking up, the C-state returns to C0, etc.). If simply waking up the polling thread does not cause the C-state to transition to C0, etc., then a light calculation may be performed here to load the CPU. Alternatively, executing the flow triggered by the hardware interrupt will also load the CPU. For this reason, the same flow as when triggered by the hardware interrupt at point 1 (start of the flow in Figure 16) may be executed.

[0123] Furthermore, if there is a high probability that packets will arrive periodically, there is also a high probability that packets will arrive after timer t has expired. For this reason, the delay time may be reduced by transitioning to (HW interrupt at point 1) and preparing to receive packets.

[0124] Instead of waking up the polling thread, the CPU's idle state can be gradually transitioned to lower states. For example, it can transition not only to C6, but also to C6 and so on. This allows for a return from a shallow idle state rather than a deep idle state, even when the packet arrival probability is low. Even if no packets arrive, the system will enter a certain idle state, thus ensuring power saving.

[0125] Alternatively, waking up using a timer can be done by utilizing a mechanism such as timerfd(registered trademark), which receives a notification from the kernel when the timer period has elapsed.

[0126] [Hardware configuration] The server-internal delay control devices 100 and 200 according to the first and second embodiments described above are implemented by a computer 900 having a configuration such as that shown in Figure 17. Figure 17 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the server-internal delay control devices 100 and 200. Computer 900 has a CPU 901, ROM 902, RAM 903, HDD 904, communication interface (I / F: Interface) 906, input / output interface (I / F) 905, and media interface (I / F) 907.

[0127] The CPU 901 operates based on programs stored in the ROM 902 or HDD 904, and controls the various parts of the server-internal delay control devices 100 and 200 shown in Figures 1 to 3 and Figure 13. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs that depend on the computer 900's hardware.

[0128] The CPU 901 controls input devices 910, such as a mouse or keyboard, and output devices 911, such as a display, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 via the input / output interface 905 and outputs the generated data to the output devices 911. In addition to the CPU 901, a GPU (Graphics Processing Unit) or the like may also be used as a processor.

[0129] HDD904 stores programs executed by CPU901 and data used by those programs. Communication I / F906 receives data from other devices via a communication network (e.g., NW(Network)920) and outputs it to CPU901, and also transmits data generated by CPU901 to other devices via the communication network.

[0130] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program related to the desired processing from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

[0131] For example, when the computer 900 functions as the server internal delay control devices 100 and 200 configured as one device according to this embodiment, the CPU 901 of the computer 900 realizes the functions of the server internal delay control devices 100 and 200 by executing the program loaded on the RAM 903. Also, the HDD 904 stores the data in the RAM 903. The CPU 901 reads and executes the program related to the target process from the recording medium 912. In addition, the CPU 901 may read the program related to the target process from another device via the communication network (NW920).

[0132] [Application Example] (Form of arranging a polling thread in the kernel) It can be applied to a server internal delay control device that starts a thread for monitoring packet arrival using a polling model in the Kernel, like the polling thread (server internal delay control device 100) shown in FIG. 2. In this case, the OS is not limited. Also, it is not limited to being in a server virtualization environment. Therefore, the server internal delay control system can be applied to each configuration shown in FIGS. 18 and 19.

[0133] <Example of Application to VM Configuration> FIG. 18 is a diagram showing an example in which the server internal delay control system 1000B is applied to an interrupt model in a general-purpose Linux kernel (registered trademark) and server virtualization environment of a VM configuration. The same components as those in FIGS. 1, 13, and 22 are denoted by the same reference numerals. As shown in FIG. 18, in the server internal delay control system 1000B, the server internal delay control devices 100 and 200 are arranged in the Kernel 171 of the Guest OS 70, and the server internal delay control devices 100 and 200 are arranged in the Kernel 91 of the Host OS 90.

[0134] Specifically, the server includes a Host OS 90 in which a virtual machine and an external process formed outside the virtual machine can operate, and a Guest OS 70 that operates within the virtual machine. HostOS90 includes Kernel91, a Ring Buffer22 managed by Kernel91 in the memory space of the server equipped with HostOS90, a receive list186 (Figure 2) that registers network device information indicating which device the hardware interrupt (hardIRQ) from NIC11 belongs to, a kernel thread vhost-net module221, a tap device222 which is a virtual interface created by Kernel91, and a virtual switch (br)223.

[0135] Kernel91 includes server-internal delay control devices 100 and 200. Kernel91 transmits packets to virtual machine 30 via tap device 222.

[0136] On the other hand, GuestOS70 includes Kernel171, a Ring Buffer52 managed by Kernel171 in the memory space of the server equipped with GuestOS70, a receive list186 (Figure 2) that registers network device information indicating which device the hardware interrupt (hardIRQ) from NIC11 belongs to, and Socket75, which is an interface for inter-process communication between Kernel171 and other components.

[0137] Kernel171 includes server-internal delay control devices 100, 200 and a protocol processing unit 74 that performs protocol processing on packets that have been pruned. Kernel171 transmits packets to packet processing APL1 via protocol processing unit 74.

[0138] By doing so, in a virtual server configuration system using VMs, packet forwarding can be performed with reduced latency within the server, regardless of whether the HostOS90 or GuestOS70 operating system is used, without modifying the APL.

[0139] <Example of application to container configuration> FIG. 19 is a diagram showing an example in which the in-server delay control system 1000C is applied to an interrupt model in a server virtualization environment with a container configuration. The same components as those in FIGS. 1 and 18 are denoted by the same reference numerals. As shown in FIG. 19, the in-server delay control system 1000C includes a Host OS 90 and has a container configuration in which the OS is replaced with a Container 211. The Container 211 has a vNIC (virtual NIC) 212.

[0140] In a system with a virtual server configuration such as a container, packet transfer can be performed with reduced delay in the server without modifying the APL. As described above, the form of arranging the polling thread in the kernel has been explained. Next, the form of arranging the polling thread in the user space will be explained.

[0141] (Form of arranging the polling thread in the user space) As shown in FIG. 3, it can be applied to a configuration example in which a polling thread (in-server delay control device 100) is arranged in the User space. In this case, the OS is not limited. Also, it is not limited to being under a server virtualization environment. Therefore, the in-server delay control system can be applied to each configuration shown in FIGS. 20 and 21.

[0142] (Example of application to the VM configuration) FIG. 20 is a diagram showing an example in which the in-server delay control system 1000D is applied to an interrupt model in a server virtualization environment with a general-purpose Linux kernel (registered trademark) and a VM configuration. The same components as those in FIGS. 1, 13, and 18 are denoted by the same reference numerals. As shown in Figure 20, the server-internal delay control system 1000D includes a Host OS 20 on which virtual machines and external processes formed outside the virtual machines can operate, and the Host OS 20 has a Kernel 21 and a Driver 23. Furthermore, the server-internal delay control system 1000D includes a NIC 11 of hardware connected to the Host OS 20, polling threads (server-internal delay control devices 100, 200) located in User space 60, a virtual switch 53, a Guest OS 1 (50) running within a virtual machine, and polling threads (server-internal delay control devices 100, 200) connected to the Host OS 20 and located in User space 60.

[0143] By doing so, in a virtual server configuration system using VMs, packet forwarding can be performed with reduced latency within the server, without modifying the APL, regardless of whether the HostOS20 or GuestOS1(50) is the OS.

[0144] <Example of application to container configuration> Figure 21 shows an example of applying the server-internal delay control system 1000E to an interrupt model in a containerized server virtualization environment. Components identical to those in Figures 1, 13, and 20 are denoted by the same reference numerals. As shown in Figure 21, the server-internal delay control system 1000E has a container configuration in which the Guest OS 50 in Figure 20 is replaced with Container 211. Container 211 has a vNIC (virtual NIC) 212.

[0145] In systems with virtual server configurations such as containers, packet forwarding can be performed with reduced latency within the server without modifying the APL (Application Program Line).

[0146] <Example of application to bare-metal configuration (non-virtualized configuration)> The present invention can be applied to a system with a non-virtualized configuration such as a bare-metal configuration. In a system with a non-virtualized configuration, packet transfer can be performed with reduced server latency without modifying the APL.

[0147] <Scale in / out> When there is a large amount of traffic and multiple NIC devices or NIC ports are used, multiple polling threads can be associated with them and scaled in / out while controlling the HW interrupt frequency.

[0148] <Expansion technology> When the number of traffic flows increases, the present invention can scale out for network load by increasing the number of CPUs assigned to the packet arrival monitoring thread in cooperation with RSS (Receive-Side Scaling) that can process inbound network traffic with multiple CPUs.

[0149] <Application to PCI device I / O such as an accelerator> Although NIC (Network Interface Card) I / O has been exemplified, this technology is also applicable to the I / O of PCI devices of accelerators (FPGA / GPU, etc.). In particular, it can be used for polling when receiving a response of an offloading result to an FEC (Forward Error Correction) accelerator in vRAN.

[0150] <Application to processors other than CPUs> The present invention is similarly applicable to processors such as GPUs / FPGAs / ASICs (application specific integrated circuits) in addition to CPUs when they have an idle state function.

[0151] [Effect] As described above, the server-internal delay control device 100 (see Figures 1 and 2) is located in the OS kernel space and starts a thread that monitors packet arrivals using a polling model. It includes a packet arrival monitoring unit 110 that polls a receive list 186 which registers network device information indicating which device a hardware interrupt (hardIRQ) from the interface unit (NIC 11) belongs to; a packet harvesting unit 120 that, if a packet has arrived, refers to the packet held in the ring buffer 72 and performs harvesting by deleting the corresponding queue entry from the ring buffer; a sleep management unit 130 that puts the polling thread to sleep if no packets arrive for a predetermined period and wakes the polling thread from sleep using a hardware interrupt (hardIRQ) when a packet arrives; and an idle state recovery control unit 150 that periodically wakes the thread when it is asleep, or wakes the thread immediately before a packet arrives in accordance with the timing of the packet's arrival.

[0152] In this way, excessive HW interrupts caused by putting the polling thread to sleep are suppressed, reducing power consumption while minimizing latency within the server and enabling packet forwarding. The server latency control device 100 prevents the CPU from falling into a deep idle state due to LPI (C-state) when the packet non-arrival time becomes prolonged by performing periodic wake-up / pre-wake-up timed to coincide with packet arrival timing, thereby suppressing the increase in latency time associated with recovering from a deep idle state (low latency method 1).

[0153] Furthermore, since packet arrival monitoring and reception processing are performed using a polling model, softIRQ contention does not occur, and latency can be reduced. Also, when a packet arrives during sleep, a polling thread is created using a high-priority hardIRQ, so the overhead caused by sleep can be suppressed as much as possible (low latency, part 2).

[0154] Furthermore, while no packets are arriving, the polling thread sleeps and the CPU frequency is set lower, thus suppressing the increase in power consumption caused by busy polling (power saving).

[0155] In summary, this server-internal packet forwarding method / system for delivering packets arriving at the NIC to the application achieves both low latency and power savings simultaneously. In particular, when the time spent without a packet arrives becomes long, the problem of the CPU core falling into a deep idle state and experiencing long latency upon recovery can be avoided by performing periodic wake-up / wake-up considering the timing of packet arrival. This ensures low latency. Furthermore, the present invention can be applied to systems like NAPI and KBP, which have polling threads within the kernel.

[0156] Furthermore, the server-internal delay control device 100 (see Figures 1 and 3) is located in user space and launches a thread that monitors packet arrivals using a polling model. The device includes a packet arrival monitoring unit 110 that polls packets arriving from the interface unit (NIC 11), a packet harvesting unit 120 that, if a packet has arrived, refers to the packet held in the ring buffer (Ring Buffer 72) and performs harvesting by deleting the corresponding queue entry from the ring buffer, a sleep management unit 130 that puts the polling thread to sleep if no packets arrive for a predetermined period and wakes the polling thread from sleep using a hardware interrupt (hardIRQ) when a packet arrives, and an idle state recovery control unit 150 that periodically wakes the thread when it is asleep, or wakes the thread immediately before a packet arrives in accordance with the timing of the packet's arrival.

[0157] In this way, excessive HW interrupts caused by putting the polling thread to sleep are suppressed, reducing power consumption while minimizing server latency and enabling packet forwarding. The server latency control device 100 (see Figures 1 and 3) can suppress the increase in latency associated with recovery from a deep idle state, as is the case with DPDK where the polling thread is in user space, thereby guaranteeing low latency.

[0158] Furthermore, the Guest OS (GuestOS70) (see Figure 18) (GuestOS180) (see Figure 19) running within the virtual machine includes a kernel (Kernel171), a ring buffer (Ring Buffer72) (see Figure 18) managed by the kernel in the memory space of the server equipped with the Guest OS, a packet arrival monitoring unit 110 that monitors the arrival of packets from the interface unit (NIC11), a packet harvesting unit 120 that, if a packet has arrived, refers to the packet held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer, and a protocol processing unit that performs protocol processing on the harvested packets. The kernel also includes server-internal delay control devices 100,200 that launch threads to monitor packet arrival using a polling model. The server-internal delay control devices 100,200 put the polling thread to sleep if no packets arrive for a predetermined period of time, and when a packet arrives, this thread (polling The system is characterized by comprising: a sleep management unit 130 that wakes the thread from sleep via a hardware interrupt (hardIRQ); and an idle state recovery control unit 150 that periodically wakes the thread during sleep, or wakes the thread immediately before the arrival of a packet in accordance with the timing of the packet's arrival.

[0159] By doing so, in a virtual server configuration system using VMs, it is possible to suppress the increased latency associated with recovering from a deep idle state for servers equipped with Guest OS (GuestOS70), thereby guaranteeing low latency.

[0160] Furthermore, a Host OS (HostOS90) (see Figure 18) (HostOS20) (see Figures 20 and 21) on which virtual machines and external processes formed outside the virtual machine can run, along with a kernel (Kernel91), manages a ring buffer (Ring) in the memory space within the server equipped with the Host OS. The system includes a Buffer 72 (see Figure 22), a packet arrival monitoring unit 110 that monitors the arrival of packets from the interface unit (NIC 11), a packet harvesting unit 120 that, if a packet has arrived, refers to the packet held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer, and a tap device 222 (see Figure 18), which is a virtual interface created by the kernel (Kernel 21). Within the kernel, there are server-internal delay control devices 100,200 that start threads to monitor packet arrival using a polling model. The server-internal delay control devices 100,200 include a packet arrival monitoring unit 110 that polls the poll list, a packet harvesting unit 120 that, if a packet has arrived, refers to the packet held in the ring buffer (Ring Buffer 72) (see Figure 22) and performs harvesting by deleting the corresponding queue entry from the ring buffer (Ring Buffer 72), and a thread (polling) if a packet has not arrived for a predetermined period of time. The system is characterized by comprising: a sleep management unit 130 that puts a polling thread into sleep mode and wakes it up via a hardware interrupt (hardIRQ) when a packet arrives; and an idle state recovery control unit 150 that periodically wakes the thread when it is asleep, or wakes the thread immediately before a packet arrives in accordance with the timing of the packet's arrival.

[0161] By doing so, in a virtual server configuration system using VMs, it is possible to reduce power consumption and suppress the increased latency associated with recovering from a deep idle state without modifying the APL, thereby guaranteeing low latency.

[0162] In the server-internal delay control device 200 (see Figure 13), the idle state recovery control unit 150 is characterized by comprising a management unit 160 that selects either a periodic wake-up control that wakes up the thread periodically during sleep, or a pre-wake-up control that wakes up the thread immediately before the arrival of a packet in accordance with the timing of the packet's arrival.

[0163] By doing so, the management unit 160 can select periodic wake-up control, which wakes up the polling thread periodically, when the packet arrival timing cannot be predicted, and pre-wake-up control, which wakes up the polling thread in advance using a timer, when the packet arrival timing can be predicted, such as in a RAN, thereby expanding the scope of application or implementation. In the first and second embodiments, the conventional method can also be selected, so it can be applied generally to existing systems without modification. Here, periodic wake-up can be applied when the packet arrival timing cannot be predicted, and pre-wake-up can maintain the deepest possible LPI (C-state) for as long as possible, thereby improving the effectiveness of sleep control and further achieving both low latency and power saving.

[0164] In the embodiments described above, we explained cases where the polling thread is located within the kernel, as in NAPI and KBP. However, it is also possible to adopt a configuration where the polling thread is located in user space, as in DPDK (see Figures 3 and 21).

[0165] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0166] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software that allows the processor to interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or an optical disc. [Explanation of symbols]

[0167] 1. Packet Processing APL (Application Program) 10 HW 11. NIC (Physical NIC) (Interface Section) 20.90 Host OS (OS) 22.72 Ring Buffer 51,70 Guest OS(OS) 60 user space 70 Guest OS 74 Protocol Processing Unit 86,186 receive list (Paul list) 90 Host OS 91,171 Kernel 100,200 Server-internal delay control devices (polling threads) 110,210 Packet Arrival Monitoring Unit 120 Packet harvesting unit 130 sleep management department 140 CPU frequency / CPU idle setting section 150 Idle State Recovery Control Unit 160 Management Department 220 Traffic Measurement Unit 211 Container 1000, 1000A, 1000B, 1000C, 1000D Server-based delay control system

Claims

1. A server-internal delay control device that is located in the OS kernel space and performs data processing by launching a thread that monitors data arrival using a polling model, A data arrival monitoring unit monitors a pole list that registers information about the network device indicating which device the hardware interrupt from the interface unit belongs to, If data has arrived, the data harvesting unit refers to the data held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer. A sleep management unit that puts the aforementioned thread to sleep and wakes the thread from sleep via a hardware interrupt when data arrives, The system includes a management unit that, if the timing of data arrival can be predicted, performs pre-wake control to wake up the thread immediately before the data arrives in accordance with the timing of the data arrival. A server-internal delay control device characterized by the following:

2. A server-internal delay control device, The Guest OS running within the virtual machine, The kernel and, In the memory space of the server equipped with the Guest OS, the ring buffer managed by the kernel and A data arrival monitoring unit that monitors the arrival of data from the interface unit, If data has arrived, the data harvesting unit refers to the data held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer. It has a protocol processing unit that performs protocol processing on the data after harvesting, The kernel includes the server-internal delay control device which launches a thread to monitor data arrival using a polling model. The aforementioned server-internal delay control device is A sleep management unit that puts the aforementioned thread to sleep and wakes the thread from sleep via a hardware interrupt when data arrives, The system includes a management unit that, if the timing of data arrival can be predicted, performs pre-wake control to wake up the thread immediately before the data arrives in accordance with the timing of the data arrival. A server-internal delay control device characterized by the following:

3. A server-internal delay control device, A host OS capable of running a virtual machine and external processes formed outside the virtual machine, The kernel and, In the memory space of the server equipped with the Host OS, the ring buffer managed by the kernel and A data arrival monitoring unit that monitors the arrival of data from the interface unit, If data has arrived, the data harvesting unit refers to the data held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer. The kernel includes a tap device which is a virtual interface created by the kernel, The kernel includes the server-internal delay control device which launches a thread to monitor data arrival using a polling model. The aforementioned server-internal delay control device is A data arrival monitoring unit monitors a pole list that registers information about the network device indicating which device the hardware interrupt from the interface unit belongs to, If data has arrived, the data harvesting unit refers to the data held in the ring buffer and performs harvesting by deleting the corresponding queue entry from the ring buffer. A sleep management unit that puts the aforementioned thread to sleep and wakes the thread from sleep via a hardware interrupt when data arrives, The system includes a management unit that, if the timing of data arrival can be predicted, performs pre-wake control to wake up the thread immediately before the data arrives in accordance with the timing of the data arrival. A server-internal delay control device characterized by the following:

4. The aforementioned management unit predicts the timing of data arrival using the data arrival interval in the RAN (Radio Access Network). A server-internal delay control device according to any one of claims 1 to 3.

5. A server-internal delay control method for a server-internal delay control device that is located in the OS kernel space and performs data processing by launching a thread that monitors data arrival using a polling model, The aforementioned server-internal delay control device is The steps include monitoring a pole list that registers information about the network device indicating which device the hardware interrupt from the interface unit belongs to, If data has arrived, the step of performing harvesting by referring to the data held in the ring buffer and removing the corresponding queue entry from the ring buffer, The steps include putting the aforementioned thread to sleep and waking the thread from sleep via a hardware interrupt when data arrives, If the timing of data arrival can be predicted, the following steps are performed:

1. Perform pre-wake control to wake up the thread immediately before the data arrives, in accordance with the timing of the data arrival. A server-internal delay control method characterized by the following:

6. A server-internal delay control method for a server-internal delay control device that is located in user space and launches a thread that monitors data arrival using a polling model, The aforementioned server-internal delay control device is A step of monitoring the arrival of data from the interface unit, If data has arrived, the step of performing harvesting by referring to the data held in the ring buffer and removing the corresponding queue entry from the ring buffer, The steps include putting the aforementioned thread to sleep and waking the thread from sleep via a hardware interrupt when data arrives, If the timing of data arrival can be predicted, the following steps are performed:

1. Perform pre-wake control to wake up the thread immediately before the data arrives, in accordance with the timing of the data arrival. A server-internal delay control method characterized by the following:

7. A program for causing a computer to function as an in-server delay control device according to any one of claims 1 to 4.

Images