Network interface and buffer control method thereof

Abstract

To suppress deterioration in performance due to a memory neck, in a network interface.SOLUTION: A network interface includes a processor, a memory, and a cache accessed by the processor ahead of the memory. The processor secures a plurality of buffers for storing transmission data in the memory and determines the presence or absence of buffer allocation to new transmission data on the basis of the buffer amount in use. The network interface sets the maximum value of the number of simultaneously-usable buffers in the plurality of buffers.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to a network interface. [Background technology]

[0002] The information system and storage system may, for example, be composed of a server system on which user applications etc. run, a storage controller that manages the stored data and improves reliability, and a drive box containing multiple drives that store and hold data used by the server system.

[0003] The storage system receives a request from the server system, and in the case of a read request, for example, the storage controller reads data from a drive in a drive box and transfers the data to the server system.

[0004] Traditionally, the front-end network of storage controllers, especially in enterprise products, has been dominated by FC (Fibre Channel) networks, while the back-end network has been dominated by SAS (Serial Attached SCSI) networks. Drive boxes have typically been configured as JBOD (Just a Bunch Of Disks), with multiple SAS / SATA drives for storing data installed in the slots of the drive box.

[0005] In recent years, in order to improve the performance of storage systems, the adoption of high-performance flash drives such as Solid State Drives (SSDs) has progressed, and the Non-Volatile Memory Express (NVMe) protocol, which is optimal for accessing flash drives, has been standardized. Furthermore, the NVMe over Fabrics (NVMe-oF) standard has emerged, which allows the NVMe protocol to be used over Internet Protocol (IP) to enable high scalability of drive connections in back-end networks.

[0006] The idea of ​​End-to-End-NVMe-oF is also spreading as NVMe-oF is being applied to front-end networks, where all connections from the server system via the storage controller to the drive box are made NVMe-oF. Drive boxes equipped with NVMe-oF-compatible flash drives (FBOF: Fabric-attached Bunch of Flash) and Composable Storage, which connects multiple storage controllers via a network, are emerging, and the use of IP connections is becoming more widespread in the enterprise storage field.

[0007] With the increase in network speeds and the emergence of new protocols (NVMe-oF, etc.), enterprise storage systems are also required to have high-speed network interfaces and support multiple protocols. Conventionally, multiple protocols have been supported by using the CPU on the storage controller or by offloading each protocol to dedicated hardware. For example, U.S. Patent No. 9,100,349 discloses a technology that supports multiple protocols in a single device using a NIC (Network Interface Card) equipped with both Ethernet and FC protocol chips.

[0008] However, using the CPU on the storage controller has the problem that CPU resources are consumed by protocol processing. Channel Boards (CHBs) equipped with ASICs dedicated to protocol processing require the development of dedicated ASICs for each protocol, making it difficult to provide timely protocol support.

[0009] In response to this, network interface devices are also changing, and SmartNICs have appeared, which incorporate general-purpose processors and memory, run an operating system, and run software on top of that to process network protocols. With SmartNICs, for example, it is possible to run the same operating system as that which runs on a server system, and to run the software protocol stacks and applications used there. Because the processing can be implemented in software, it is possible to flexibly handle multiple protocols, rapid support for new protocols, and updates to protocol processing. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] U.S. Patent No. 9,100,349 Summary of the Invention [Problem to be solved by the invention]

[0011] By implementing protocol processing in software using a SmartNIC, a NIC equipped with a general-purpose CPU and memory, it is possible to flexibly support multiple protocols while offloading the protocol processing load from the storage controller.

[0012] However, due to the constraints of the board area required for mounting a SmartNIC on a storage controller and the power limitations of the storage controller interface, there are limitations on the type, number, and wiring of memory chips that can be mounted.

[0013] For example, protocols such as iSCSI and NVMe TCP transfer data in TCP / IP packets, store the packets in the SmartNIC memory, and require multiple memory accesses, so it may not be possible to achieve a configuration that achieves the required memory performance. As a result, memory may become a performance bottleneck and the desired performance may not be achieved. [Means for solving the problem]

[0014] A network interface according to a representative embodiment of the present invention includes a processor, a memory, and a cache that is accessed by the processor before the memory, and the processor reserves multiple buffers in the memory for storing transfer data, determines whether or not to allocate a buffer for new transfer data based on the amount of buffers in use, and sets a maximum number of buffers that can be used simultaneously among the multiple buffers. Effect of the Invention

[0015] According to a representative embodiment of the present invention, it is possible to suppress performance degradation due to a memory bottleneck in a network interface. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiment. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 illustrates an example of the configuration of a network interface. [Diagram 3] FIG. 4 is a diagram showing a program configuration of a network interface. [Figure 4] FIG. 13 is a diagram illustrating an example of a program relationship of a storage interface. [Diagram 5] FIG. 2 is a diagram illustrating an outline of a process in which a network interface transmits and receives commands and data to and from a host. [Figure 6A] FIG. 2 is a diagram illustrating some aspects of data transfer. [Figure 6B] FIG. 2 is a diagram illustrating some aspects of data transfer. [Figure 7] An example of the configuration of a buffer list is shown below. [Figure 8A] This indicates the buffer list management information contained in the buffer list. [Figure 8B] Indicates the information held by the buffer entry. [Figure 9] 13 shows an example of the configuration of a buffer management table. [Figure 10] 1 shows an example of a process flow of a target network interface. [Figure 11] 13 shows an example of a process flow of a network interface of an initiator. [Figure 12] 13 shows a flowchart of an example of a buffer control process. [Figure 13] 13 shows a flowchart of an example of a buffer list determination process. [Figure 14] 13 is a flowchart of an example of a buffer control process. [Figure 15] 13 shows a flowchart of an example of processing in a buffer list determination step. [Figure 16] 13 shows a flowchart of an example of a maximum active number determination process. [Figure 17] 13 shows a flowchart of another example of a maximum active number determination process. [Figure 18] 13 shows a flowchart of another example of a maximum active number determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment will be described with reference to the drawings. Note that the following description and drawings are merely examples for explaining the present invention, and are omitted or simplified as appropriate for clarity of explanation. The present invention can be embodied in various other forms, and each component may be singular or plural unless otherwise specified.

[0018] Furthermore, the embodiments described below do not limit the invention according to the claims, and not all of the combinations of elements described in the embodiments are necessarily essential to the solution of the invention.

[0019] In the following description, various information may be described using expressions such as "table," "list," and "queue," but the various information may be expressed in data structures other than these, and in order to indicate that the information is not dependent on the data structure, "xxx table," "xxx list," "xxx queue," etc. may be referred to as "xxx information," etc. In the following description, when describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0020] In the following description, when there are multiple components having the same or similar functions, they are basically described with the same reference numerals, but even if the functions are the same, the means for realizing the functions may be different. Furthermore, the embodiments described below may be implemented as software running on a general-purpose computer, or as dedicated hardware or a combination of software and hardware.

[0021] In addition, in the following explanation, processing may be described using a "program" as the subject, but since a program is executed by a processor (e.g., CPU: Central Processing Unit) to perform a specified process while appropriately using storage resources (e.g., memory) and / or interface devices (communication ports), the subject of the processing may also be described as the processor.

[0022] The processing described with a program as the subject may be processing performed by a computer having a processor (e.g., a computing host, a storage device). In the following description, the term "controller" may refer to a processor or a hardware circuit that performs all or part of the processing performed by the processor.

[0023] The program may be installed on each computer from a program source (e.g., a program distribution server or a computer-readable storage medium), in which case the program distribution server includes a CPU and storage resources, and the storage resources further store the distribution program and the program to be distributed, and the CPU of the program distribution server may distribute the program to be distributed to other computers by executing the distribution program.

[0024] In the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0025] In the following description, a storage drive or simply a drive means a physical storage device, and may typically be a non-volatile storage device (e.g., an auxiliary storage device). The drive may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). Different types of drives may be mixed in the storage system.

[0026] Also, in the following description, a drive has a VOL, where "VOL" is an abbreviation for volume and may be a physical storage device or a logical storage device. A VOL may be an actual VOL (RVOL) or a virtual VOL (VVOL). An "RVOL" may be a VOL based on a physical storage resource (for example, one or more RAID groups) of the storage system having that RVOL.

[0027] A "VVOL" may be any of an Externally Connected VOL (EVOL), a Capacity Expanded VOL (TPVOL), and a Snapshot VOL. An EVOL is based on the storage space (e.g., a VOL) ​​of an external storage system and may be a VOL that complies with storage virtualization technology. A TPVOL is composed of multiple virtual areas (virtual storage areas) and may be a VOL that complies with capacity virtualization technology (typically Thin Provisioning).

[0028] In the following explanation, a VOL recognized by a host (a VOL provided to a host) is called an "LDEV." In the following explanation, an LDEV is a TPVOL (or RVOL), and a pool is a TP pool. However, the features of this disclosure can also be applied to storage devices that do not employ capacity expansion technology (Thin Provisioning).

[0029] A "pool (POOL)" is a logical storage area (for example, a collection of multiple pool VOLs) and may be prepared for each purpose. For example, the pool may be a TP pool. The TP pool may be a storage area composed of multiple pages (physical storage areas). If a page is not assigned to the virtual area (the virtual area of ​​the TPVOL) to which the address specified in the write request received from the host computer (hereinafter, host) belongs, the storage controller assigns a page from the TP pool to that virtual area (write destination virtual area) (even if a page has already been assigned to the write destination virtual area, a page may be newly assigned to the write destination virtual area). A "pool VOL" may be a VOL that is a component of a pool. The pool VOL may be an RVOL or an EVOL.

[0030] In addition, in the following description, a VOL may be a "Logical Unit (hereinafter, LU)" in SCSI or a "Name Space (hereinafter, NS)" in NVMe.

[0031] In the following description, "RAID" is an abbreviation for Redundant Array of Inexpensive Disks. A RAID group is made up of multiple drives (typically drives of the same type) and stores data according to the RAID level associated with the RAID group. A RAID group may also be called a parity group. A parity group may be, for example, a RAID group that stores parity.

[0032] A network interface device (hereinafter also simply referred to as a network interface) according to an embodiment of the present specification may be implemented in an information processing system including a server system and a storage system. The storage system may include a storage controller and a drive box. The network interface may include, for example, a general-purpose processor, a memory, a network controller, and an interface with a host system.

[0033] A configuration using a general-purpose processor and memory capable of implementing software-based protocol processing solves the problem of having to update hardware every time new protocols such as NVMe-oF are added in addition to iSCSI. For example, in order to quickly respond to changes due to switching from iSCSI to NVMe-oF, changes due to NVMe-oF specification updates, and even changes to new protocols, the network interface software is replaced using a general-purpose processor, memory, etc. to keep up with the changes. The network interface of one embodiment of this specification supports multiple protocol processes simultaneously.

[0034] In order to reduce memory accesses in protocol processing, a network interface according to one embodiment of this specification can adopt some or all of the following methods for data buffer control that stores data that is the dominant factor in memory accesses during protocol processing.

[0035] The network interface uses buffers of multiple sizes. The buffer size is set according to the transfer data size of the supported protocol, and for example, multiple data buffers are allocated for each supported transfer data size. The data buffers are allocated in a list structure, for example, at initialization (startup).

[0036] For example, when a network interface starts up, it allocates multiple buffers of different sizes, such as 8 KB and 256 KB. This makes it possible to use different buffers for short and long data lengths while reducing the buffer allocation load during I / O. When many short data transfer requests are received, the amount of memory used is reduced, and when a long data transfer request is received, the load of looking up the list is reduced by not using many buffers.

[0037] A network interface according to an embodiment of this specification selects a recently used area when selecting a data buffer. Data buffers are managed in a list, and the most recently released buffer is used in the next process. This makes it possible to localize the range of access to the data buffer and increase the cache hit rate. The transferred data hits the cache, reducing the number of memory accesses, thereby resolving the issue of performance degradation caused by memory bottlenecks.

[0038] The network interface according to an embodiment of this specification controls the command processing that can be started based on the difference between the cache capacity and the total amount of active buffer usage. In order to prevent data from overflowing from the cache as much as possible, the number of buffers used for simultaneously processing data transfer is limited. This increases the cache hit frequency, reduces memory access, and prevents performance degradation due to memory bandwidth pressure.

[0039] 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present specification. The information processing system includes one or more server systems 100 and a storage system. The storage system includes one or more storage devices 101 and one or more drive boxes 105.

[0040] The storage device 101 includes one or more storage controllers 102. In the configuration example of Fig. 1, each storage device 101 includes two storage controllers 102. The storage controllers 102 are connected to one or more server systems 100 via a front-end network 106.

[0041] A drive box 105 is equipped with one or more drives, and is connected to one or more storage controllers 102 via a back-end network 107. The storage controller 102 is connected to other storage controllers 102 at medium or long distances via an external network 109, in addition to being connected to a short distance via an inter-storage controller network 108.

[0042] The server system 100 is a host machine on which user applications and the like run, and has one or more processors, and is configured to include one or more storage devices, such as memory and auxiliary storage devices. For example, a database or Web service runs on the server system 100, and the server system 100 writes and reads data created by these to and from the storage controller 102 via a network interface 104. The server system 100 is also connected to the storage controller 102 via a front-end network 106, and has a network interface 104 as its interface device. The server system 100 may be configured from a group of multiple servers, each of which may have a network interface 104 and be connected to the storage controller 102 or another server system 100.

[0043] In order to provide storage functions to the server system 100, the two storage controllers 102 in the storage device 101 configure a redundant controller. Each of the storage controllers 102 includes one or more processors and one or more storage devices. The storage controllers 102 in the storage device 101 have the same configuration.

[0044] Each storage controller 102 has one or more processors, and the core of each processor instructs the transfer of data stored in the corresponding drive box 105 in response to a read command or a write command from the server system 100. The memory of the storage controller 102 is composed of a semiconductor memory such as a Synchronous Dynamic Random Access Memory (SDRAM). The memory may be composed of a combination of a volatile memory and a non-volatile memory such as a Storage Class Memory (SCM).

[0045] The memory serves as the processor's main memory and stores execution programs (storage control programs, etc.), management tables referenced by the processor, etc. The memory is also used as a disk cache (cache memory) for the storage controller 102. The storage controller 102 has a network interface 104 as an interface device for the drive box 105. The network interface 104 communicates with the drive box 105 information relating to processing related to storage processing such as data transfer and data copy instructed by the server system 100.

[0046] The drive box 105 is equipped with a plurality of drives such as SSDs and HDDs, and includes an internal switch, as well as a processor and memory used for transfer processing, to connect the plurality of drives to the storage controller 102. Data generated by the server system 100 is received via the storage controller 102, and is stored and held.

[0047] To ensure the availability of data stored in the drive box 105, a RAID may be formed between the built-in drives, or a RAID may be formed between multiple drive boxes 105. The drive box 105 is also connected to the storage controller 102 via a back-end network 107, and has a network interface 104 as its interface device.

[0048] The network interface 104 is a device that is installed in each of the server system 100, the storage controller 102, and the drive box 105, and serves as a connection interface between various devices and various networks.

[0049] The network interface 104 may be, for example, a SmartNIC. Various functions of the SmartNIC are implemented using a general-purpose processor equipped with the SmartNIC and a partial hardware offload engine. The SmartNIC may also be configured using an FPGA (Field Programmable Gate Array), in which case each function is realized on the FPGA. As another form, the entire network interface 104 may be configured as dedicated interface hardware implemented in hardware. Details of the network interface 104 will be described later.

[0050] The front-end network 106 is a storage area network that connects between the storage controller 102 and the server system 100, and may be, for example, an IP network such as iSCSI or NVMe-oF (NVMe over Fabrics). The back-end network 107 is a network that connects between the storage controller 102 and the drive box 105, and may be, for example, an IP network such as iSCSI or NVMe-oF.

[0051] The storage controller network 108 is a network used for redundancy of the storage controllers 102, and is configured with a broadband interconnect. Using this network, write data is duplicated and metadata is shared, so that even if one storage controller 102 is blocked due to maintenance or a failure, the other storage controller 102 can continue storage processing.

[0052] The external network 109 is a wide area network (WAN) or a local area network (LAN), for example, a network with a data link layer of Ethernet (registered trademark), an internet layer of Internet Protocol, and a transport layer of TCP or UDP, etc., and performs communication using a protocol data unit (PDU) of iSCSI or NVMe-oF. The network can take the form of an internet line or a dedicated line. Communication delay increases depending on the distance, and if the network devices are not configured only with lossless devices, packet loss is expected to occur, although the occurrence rate differs depending on the above line type.

[0053] The information system and storage system may include systems other than those shown here. For example, each network may be connected to a network device such as a switch or a router, or to a device for monitoring or maintenance. Also, a configuration may be used in which a storage service on a public cloud is connected via an external network 109.

[0054] The network interface 104 of the server system 100 has initiator software for iSCSI or NVMe-oF to read and write data in the drive box 105 via the storage controller 102. In contrast, the network interface 104 of the storage controller has target software.

[0055] The network interface 104 of the storage controller 102 has initiator software for iSCSI or NVMe-oF to read and write data to and from the drive box 105. In contrast, the network interface 104 of the drive box 105 has target software.

[0056] Furthermore, the network interface 104 of the storage controller 102 has initiator software for iSCSI or NVMe-oF to read and write data from and to other storage devices 101. In contrast, the network interface 104 of the other storage devices 101 has target software.

[0057] Note that some of the network interfaces 104 may be NICs that do not have the functionality of a SmartNIC. For example, the NIC 104 in the network interface 100 may be configured to operate as an initiator, and the NIC 104 in the network interface 102 may be configured to operate as a target.

[0058] 2 is a diagram showing an example of the configuration of a network interface 104 according to an embodiment of the present specification. The network interface 104 is connected to a device having the network interface 104 mounted thereon, such as the storage controller 102, the internal configuration of the server system 100, or a drive box 105, via a host bus 201. The network interface 104 is connected to other devices via a network path 200, for example, by an IP network connection.

[0059] The network interface 104 includes a data processing unit 202 and a memory 208. The data processing unit 202 includes a host interface 207 for connecting to a host bus 201, and a network controller 203 for connecting to a network path 200 and processing a network protocol. The data processing unit 202 further includes a processor 204, a memory controller 209, a DMA (Direct Memory Access) controller 205, and a cache 206.

[0060] The host bus 201 is a bus that connects the network interface 104 to the storage controller 102, the server system 100, the drive box 105, etc. The host bus 201 is a broadband, high-speed interconnect, and is configured to connect to a processor of a device mounted thereon via PCIe, for example.

[0061] The host interface 207 is an interface for connecting the network interface 104 to a host system (hereinafter also simply referred to as a host) via the host bus 201. For example, if the host bus 201 is PCIe, it may include, for example, a PCIe PHY. The DMA controller 205 exchanges data between the memory 208 in the network interface 104 and the memory of the host.

[0062] The network path 200 is, for example, an IP network path, and may take the form of a WAN, LAN, or SAN (Storage Area Network). The network interface 104 communicates via one network path 200, or two or more network paths 200 in consideration of redundancy.

[0063] The network controller 203 is an interface for connecting the network interface 104 to the network path 200. The network controller 203 executes, for example, physical layer processing such as PHY, and stateless processing of the data link layer, the internet layer, and the transport layer. The network controller 203 performs, for example, checksum and frame processing.

[0064] The network controller 203 supports, for example, Ethernet, IP, TCP, UDP, etc. Furthermore, it may include an offload engine for IPsec (Internet Security Protocol), TLS (Transport Layer Security), DIF (Data Integrity Field), etc. Also, it has a configuration compatible with connection with optical cables, copper cables, etc.

[0065] The processor 204 is, for example, a general-purpose processor, and executes, for example, an operating system that is also used in the server system 100. The processor 204 further executes other software to perform processes such as protocol processing, command processing, and management of the network interface 104. The processor 204 can have any configuration, and can include, for example, one or more CPUs or MPUs (Micro Processing Units), and one or more cores 241.

[0066] The memory 208 is composed of, for example, a semiconductor memory such as an SDRAM, and may be configured in combination with a non-volatile memory such as an SCM. The memory 208 serves as the main memory for the processor 204 and stores execution programs (instruction codes for protocol processing and command processing) and management tables referenced by the processor. The memory 208 is also used as a buffer for commands and data transmitted and received from the network. Furthermore, the memory 208 provides a queuing interface with the network controller 203 and the host interface 207, and stores queue descriptors, indexes, etc.

[0067] The memory controller 209 is an interface for controlling the memory when reading and writing data from and to the memory 208. The memory controller 209 may be incorporated in the processor 204, the data processing unit 202, or the network interface 104, for example.

[0068] The cache 206 temporarily stores data between the memory 208 and the processor 204. The processor 204 can access the cache 206 faster than the memory 208. Data read from the memory 208 is stored in the cache 206. The processor 204 accesses the cache 206 before the memory 208 to read data (including commands). The cache 206 can have a hierarchical structure. From the hierarchy closer to the processor, it is called an L1 cache, an L2 cache, and so on. The processor 204 and the DMA controller 205 ensure coherency (consistency) with respect to the cache 206.

[0069] It should be noted that the information system and storage system may include other components in addition to those shown here, such as additional modules and interfaces for monitoring and maintenance, and non-volatile storage devices for storing operating systems and software programs running on the network interface 104.

[0070] 3 is a diagram showing a program configuration of a network interface according to an embodiment of the present specification. In order to realize software-based, changeable processing functions, the network interface 104 uses a general-purpose processor 204, a cache 206, and a memory 208 to run an operating system 300, on which various processing software programs are run.

[0071] The software programs 300 to 308 are loaded into memory 208 and processed by processor 204. Some of the instructions are stored in cache 206. Note that processing parts that are more efficient when implemented by hardware, such as DIF (Data Integrity Field), CRC (Cyclic Redundancy Check), encryption, compression, hashing, and parity processing, may be implemented in hardware and controlled by software.

[0072] The operating system 300 is a basic software program that is the basis for operating the network interface, and manages the entire network interface. The operating system 300 provides a common usage environment for each piece of software that runs on the processor of the network interface. The operating system 300 may be an embedded operating system, or a general-purpose operating system that runs on a server, such as Linux (registered trademark).

[0073] The network controller driver 301 is driver software for controlling the network controller 203. The network controller driver 301 passes a parameter set for packet processing to be offloaded when generating or receiving a packet to the network controller 203. Furthermore, the network controller driver 301 passes a packet generated by the protocol processing 303 to the network controller 203 for transmission. Also, the network controller driver 301 passes a packet received from the network controller 203 to the protocol processing 303.

[0074] The host interface driver 302 is driver software for controlling the host interface 207. Communication between the network interface 104 and the host is performed via the host interface driver 302.

[0075] The protocol processing 303 cooperates with the command processing 305, the DMA control 307, and the network controller driver 301 to generate and transmit packets. It also processes received packets to extract information and data such as control information, iSCSI PDU, and NVMe-oF PDU, and passes the extracted information to the command processing 305.

[0076] The protocol processing 303 performs IP header processing at the Internet Protocol layer, TCP header processing and UDP header processing at the transport layer, and iSCSI processing and NVMe-oF processing. For example, the protocol processing 303 executes programs such as a socket program, an iSCSI initiator or target, and an NVMe-oF initiator or target.

[0077] The host queue control 304 is software for controlling a queue interface for exchanging commands with the host. The host queue control 304 manages, within the network interface 104, an internal queue of commands to be sent to the host and an internal queue of commands received from the host. The host queue control 304 stores commands to the host or commands from the host in the internal queue. It also controls the head and tail when the internal queue has, for example, a ring structure. The host queue control 304 controls the head and tail of the command queue held by the host.

[0078] Command processing 305 receives commands for controlling the network interface 104 from the host, and controls the network interface 104. It receives a network protocol processing request from the host, starts protocol processing 303, and responds to the host with the processing results of protocol processing 303. It also starts DMA control 307 for data transfer with memory reserved by the host, and performs response processing. It also performs initial settings and setting changes, software replacement for the network interface 104, and notifies the host in the event of a failure.

[0079] Buffer control 306 is software for controlling buffers that temporarily hold data in memory 208. The buffers store data received from the network and transferred to the host, or data received from the host and sent to the network. Buffer control 306 allocates, uses, and manages a group of buffers of different sizes. Buffer control 306 controls the buffers to improve the cache hit rate. This point will be described in more detail later.

[0080] The DMA control 307 performs interface processing with DMA hardware to control data transfer between, for example, a memory secured on the host side and the memory 208 on the network interface 104 .

[0081] The initialization / maintenance / fault processing 308 initializes the hardware that constitutes the network interface 104 and initializes various pieces of software. It also supports software updates for the network interface 104, detection of hardware faults and notification of the faults to the host, etc.

[0082] Fig. 4 is a diagram showing an example of the relationship of the programs of the storage interface shown in Fig. 3. An operating system 300 runs in the network interface 104, and each software program runs based on the operating system 300. Furthermore, the network interface 104 executes initial settings, maintenance such as software updates, and fault processing, etc., through initialization / maintenance / fault processing 308. Fault processing includes, for example, fault detection, and collection of dump trace information such as statistical information and error information.

[0083] The network controller driver 301 controls the network controller 203, stores transmission packets in a packet buffer of the network controller 203, and acquires received packets from the packet buffer. It also performs settings for offloading frame processing of the data link layer, the internet protocol layer, and the transport layer, and stateless processing such as checksum calculation.

[0084] In response to the operation of the network controller driver 301, the protocol processing 303 performs network protocol processing such as IP, TCP, UDP, iSCSI PDU processing, and NVMe-oF PDU processing. The protocol processing 303 processes packets received from the network controller driver 301 to extract information and data such as control information, iSCSI PDU, and NVMe-oF PDU. The received packets may not contain data. The protocol processing 303 passes the extracted information to the command processing 305. The data may be transferred to the host via a buffer controlled by the buffer control 306 or without a buffer.

[0085] Protocol process 303 includes the information obtained from command process 305 and the data sent from the host in a packet sent to the network. The data may be transferred to the network with or without going through a buffer controlled by buffer control 306. The packet sent may not contain any data.

[0086] Command processing 305 executes command processing in cooperation with host queue control 304, buffer control 306, and DMA control 307. Command processing 305 controls the host interface 207 via the host interface driver 302. Command processing 305 exchanges send commands or receive commands for network communication with the host system, such as information for generating PDUs for iSCSI or NVMe-oF, analyzed information, or the PDU itself. Commands are exchanged with the host system using a queue interface. Host queue control 304 controls this queue interface.

[0087] Command processing 305 generates a descriptor that can be processed by the host from the result of protocol processing 303, and stores it in a queue via host queue control 304. It also obtains the descriptor generated by the host system from the queue, and sets and uses each function in network interface 104, such as protocol processing 303. Note that the queue descriptor can be exchanged between the host system and network interface 104 using DMA controller 205 by DMA control 307.

[0088] 5 is a diagram for explaining an overview of the process in which the network interface 104 transmits and receives commands and data to and from a host, for example, the storage controller 102. In the following, commands exchanged with the host are also referred to as host commands.

[0089] The network interface 104 receives a packet 351 from the network and assigns its processing to one of the cores 241. The assigned core 241 executes the protocol processing and extracts information and data. In the example of Fig. 5, the extracted data is stored in a buffer 381. The core 241 generates a host command 355 to be sent to the host from the information extracted from the received packet 351.

[0090] The host command is stored in an internal transmission queue 371. An internal transmission queue 371 is provided for each core 241, and in Fig. 5, one internal transmission queue is indicated by the reference numeral 371 as an example. In the example of Fig. 5, the internal transmission queue 371 is a ring buffer (FIFO), and a new command is stored in its tail area. The head and tail of each internal transmission queue 371 are managed.

[0091] The network interface 104 transfers the host command retrieved from the Head of the internal send queue 371 to the host send queue 401 of the host 400. The host 400 is, for example, the storage controller 102 or the server system 100. The host send queue 401 is a ring buffer in the example of Fig. 5. The network interface 104 manages the Head and Tail of the host send queue 401 together with the host 400.

[0092] 5, host commands from multiple internal transmit queues 371 are transferred to one host transmit queue 401. In another example, multiple host transmit queues 401 may be prepared, and for example, the same number of host transmit queues 401 corresponding to each of the internal transmit queues 371 may be used.

[0093] The host 400 stores host commands to be transferred to the network interface 104 in a host receive queue 402. The host receive queue 402 is a ring buffer. The network interface 104, together with the host 400, manages the head and tail of the host receive queue 402.

[0094] The network interface 104 transfers the command in the host receive queue 402 to the internal receive queue 372. An internal receive queue 372 is provided for each core 241, and in FIG. 5, the internal receive queue 372 selects and transfers the host command transferred from the host. At this time, the host command may be checked by the host queue control 304 or by DMA hardware. One internal receive queue is indicated by the reference numeral 372, for example. A plurality of host receive queues 402 may be provided, and for example, the same number of host receive queues 402 corresponding to each internal receive queue 372 may be used so that the allocation process to the internal receive queues 372 is not required.

[0095] The network interface 104 transfers data stored in the buffer 381 to the host 400 in response to a host command from the host 400, and also transfers data 354 stored in the memory of the host 400 to the buffer 382. As will be described later, the buffers 381 and 382 are controlled so as to improve the cache hit rate.

[0096] Upon receiving a host command 356 to transmit a packet including data 354 to the network, the network interface 104 performs protocol processing of the data 354 in accordance with the host command 356 to generate a transmit packet 352. The network interface 104 transmits the transmit packet 352 to the network.

[0097] Figures 6A and 6B are diagrams illustrating several aspects of data transfer. Figure 6A shows several examples of data transfer using the cache 206 / memory 208 of the network interface 104, and Figure 6B shows several examples of data transfer without using the cache 206 / memory 208. The data transfer is performed using the DMA controller 205. The network interface 104 may support all of the data transfers shown in Figures 6A and 6B, or may support only some of the data transfers, for example, the data transfer in Figure 6A.

[0098] FIG. 6A shows data transfer via the cache / memory of the network interface 104. For example, data transfer of iSCSI or NVMe TCP is applicable. In FIG. 6A, data transfer 501 is executed between the host 400 and the cache 206 / memory 208 of the network interface 104. In the data transfer 501, no error check is performed, and no guarantee code is added or removed. The data transfer 501 corresponds to, for example, update data of a program running on the network interface 104, and transfer of a descriptor transferred via a queue. Data transfer 511 is executed between the host 400 and the remote site 450, via the cache 206 / memory 208 of the network interface 104. In the data transfer 511, no error check is performed, and no guarantee code is added or removed.

[0099] Data transfer 512 is performed from the remote site 450 to the host 400 via the cache 206 / memory 208 of the network interface 104. The DMA controller 205 adds a DIF (Data Integrity Field) to the data from the remote site 450 and transfers it to the host 400. Information for generating the DIF is obtained from the host 400.

[0100] Data transfer 513 is executed from the host 400 to the remote site 450 via the cache 206 / memory 208 of the network interface 104. The DMA controller 205 checks the data by referring to the DIF added to the data from the host 400, and removes the DIF from the data. The data from which the DIF has been removed is transferred to the remote site 450.

[0101] Data transfer 514 is performed between the host 400 and the remote site 450 via the cache 206 / memory 208 of the network interface 104. A DIF is attached to the data to be transferred. The DMA controller 205 performs an error check by referring to the DIF attached to the data.

[0102] Data transfer 515 is performed between the host 400 and the remote site 450 via the cache 206 / memory 208 of the network interface 104. A DIF is attached to the data to be transferred. The DMA controller 205 performs an error check by referring to the DIF attached to the data, removes the DIF, and attaches a new DIF.

[0103] 6B shows data transfer that does not go through the cache / memory of the network interface 104. For example, this corresponds to RDMA (Remote Direct Memory Access).

[0104] Data transfer 521 is executed between the host 400 and the remote site 450. No error check or addition or removal of a guarantee code is performed in the data transfer 521. Data transfer 522 is executed from the remote site 450 to the host 400. The DMA controller 205 adds a DIF (Data Integrity Field) to the data from the remote site 450 and transfers it to the host 400.

[0105] Data transfer 523 is executed from host 400 to remote site 450. DMA controller 205 performs a data check by referring to the DIF added to the data from host 400, and removes the DIF from the data. The data from which the DIF has been removed is transferred to remote site 450.

[0106] Data transfer 524 is performed between host 400 and remote site 450. A DIF is attached to the data to be transferred. DMA controller 205 performs an error check by referring to the DIF attached to the data. Data transfer 525 is performed between host 400 and remote site 450. A DIF is attached to the data to be transferred. DMA controller 205 performs an error check by referring to the DIF attached to the data, removes the DIF, and attaches a new DIF.

[0107] 7 shows an example of the configuration of a buffer list. The network interface 104 creates and manages buffer groups of different sizes in the memory 208. Each buffer group is composed of multiple buffers. The different sizes are set according to the supported data transfer sizes. For example, buffers of sizes such as 8 KB, 16 KB, 32 KB, 256 KB, etc. are secured when the network interface 104 is started.

[0108] Each buffer group is managed by a corresponding buffer list. Figure 7 shows an example of a buffer list 700A for a buffer group of size X, a buffer list 700B for a buffer group of size Y, and a buffer list 700C for a buffer group of size Z. The buffer lists are managed by the buffer control 306.

[0109] Each buffer list includes multiple buffer entries 703. In Fig. 7, one buffer entry is indicated by reference symbol 703 as an example. The buffer entry 703 is represented by a rectangle, and the number inside indicates the entry number. For example, a buffer list with a size of 8 KB manages buffers with a buffer entry count of 8 KB. The buffer entry 703 indicates information about the corresponding buffer.

[0110] 8A shows buffer list management information 750 included in each buffer list. The buffer list management information shows a buffer entry head address 751 and a buffer active number 752. The buffer entry head address 751 shows the address of the first entry in the buffer list. The buffer active number 752 shows the number of buffer entries currently in use.

[0111] 8B shows information held by the buffer entry 703. The buffer entry 703 shows a buffer entry address 761, a buffer address 762, and a next buffer entry address 763. The buffer entry address 761 shows the address of the buffer entry. The buffer address 762 shows the address in memory 208 of the buffer managed by the buffer entry 703. The next buffer entry address 763 shows the address of the buffer entry next to the buffer entry 703.

[0112] The buffer entry 703 may manage whether the buffer is active or free in a field not shown. The buffer entry head address 751 indicates the first entry of the buffer entries of the free buffer. The buffer entries of the free buffer are ordered and arranged according to the next buffer entry address 763. In this way, the buffer list selects a buffer entry starting from the first buffer entry to store new data. In this way, the allocation order of the free buffers is managed.

[0113] When the top buffer entry is selected, the buffer entry head address 751 is updated. Therefore, the entry of the buffer in use is removed from the list of the free buffer entries. As will be described later, the entry of the buffer that has been used in the data transfer and then released is returned to a predetermined position in the list of free buffer entries, the top position in the example described later.

[0114] The buffer list of multiple buffer sizes reduces the load of securing buffers during I / O, while using buffers for short data length and long data length separately, thereby reducing the amount of memory used when many short data length transfer requests are received, and reducing the number of buffers used when long data length transfer requests are received, thereby reducing the load of looking up the list.

[0115] 9 shows an example of the configuration of a buffer management table 900. The buffer management table 900 indicates information for managing buffers, and is generated by the storage controller 102. The buffer management table 900 enables the buffer group to be dynamically changed. Specifically, the buffer management table 900 indicates a buffer list 771, a buffer size 772, a number of buffers 773, a maximum active number 774, and a control type 775.

[0116] The buffer list 771 indicates an identifier of the buffer list. The buffer size 772 indicates the buffer size of the buffer group managed by the buffer list. The number of buffers 773 indicates the number of buffers managed by the buffer list, that is, the number of buffer entries. The number of buffers 773 indicates the number of buffers secured at initialization. The maximum active number 774 indicates the maximum number of buffers that can be used simultaneously in the buffer group managed by the buffer list. By changing the maximum active number, the maximum number of buffers that can be used simultaneously is dynamically controlled. The control type 775 indicates the control type of the buffer. The buffer control method is switched depending on the control type. For example, the method of calculating the allowable active number of buffers can be switched between performing the calculation for each buffer list, or performing the calculation based on the active number of all buffer lists. In addition, when transferring on the buffer size of the buffer list, the method of using multiple buffers or reserving new space can be switched.

[0117] 10 shows an example of a processing flow of the target network interface 104. The network interface 104 executes processing on a communication packet from the network to the host of the network interface 104. For example, the host is the target storage controller 102, to which the communication packet is transferred from the initiator server system 100 via the network 106.

[0118] The network controller 203 receives a communication packet from a network (S101). The network controller 203 distributes protocol processing of the received communication packet to a core of the processor 204 (S102).

[0119] The core that is assigned the protocol processing of the communication packet executes the protocol processing 303 (S103). The protocol processing 303 extracts information from the communication packet and passes it to the command processing 305.

[0120] The command processing 305 generates a host command for notifying the host of the communication protocol processing result according to the information extracted from the communication packet (S104). The notification of the processing result is, for example, notifying the storage controller 102 that a read request or a write request has been received from the network. Furthermore, the command processing 305 loads the generated host command into the internal transmission queue 371 in the network interface 104 (S105). In response to the loading of the host command, the command processing 305 updates the Tail of the internal transmission queue 371 (S106).

[0121] The host queue control 304 receives the Tail update of the internal transmit queue 371 of the core it is responsible for, and transfers the host command in the internal transmit queue 371 to the host transmit queue using the DMA controller 205 via the DMA control 307 (S107). In response to the transfer of the host command, the host queue control 304 updates the Tail of the host transmit queue 401 (S108). The network interface 104 and the host may refer to the same information for the Tail and Head information of the host transmit queue 401, or the information held by each may be transferred. Furthermore, the host queue control 304 updates the Head of the internal transmit queue (S109).

[0122] The host acquires a host command from the host transmission queue 401 and updates the Head of the host transmission queue (S110). The network interface 104 may receive and update the Head of the host transmission queue in accordance with the update on the host side, or may receive a notification from the host at a different timing. The host refers to the acquired host command and determines whether it is necessary to send the command to the network interface 104 (S111).

[0123] When the host needs to send a host command to the network interface 104 (S111: YES), the host loads the host command into the host receive queue 402 (S112) and updates the Tail of the host receive queue 402 (S113). In response to the host updating the information in the host receive queue 402, the network interface 104 also updates the information in the host receive queue 402 that it holds. For this update, the host and the network interface 104 may refer to common information, or the updated information may be transferred between them.

[0124] Upon receiving the tail update of the host receive queue 402, the host queue control 304 of one of the cores transfers the host command stacked in the host receive queue 402 to the network interface 104 by using the DMA controller 205 through the DMA control 307 (S114). For example, the cores may periodically obtain a lock to check whether the host receive queue has been updated, and if there is an update, they may perform a transfer process, or a core may be provided that is dedicated to checking and transferring updates to the host receive queue, or multiple host receive queues may be created, and if there is an update to the receive queue that the core is responsible for, a transfer process may be performed. The host queue control 304 loads the host command from the host into the internal receive queue 372 (S115). For example, the contents of the host command may be checked to determine which internal receive queue 372 to load the command into, or if there are multiple host receive queues, the command may be loaded into the corresponding internal receive queue 372.

[0125] In response to the loading of the host command into the internal receive queue 372, the host queue control 304 updates the Tail of the internal receive queue 372 (S116), and further updates the Head of the host receive queue 402 (S117). The method of updating the Head of the host receive queue 402 is as described above for updating the Tail of the host receive queue 402.

[0126] The command processing 305 acquires the host command from the internal receive queue 372 and executes the processing (S118). If data transfer using a buffer is required (S119: YES), the buffer control step S120 is executed. Details of the buffer control step S120 will be described later. The data transfer is a data transfer from the network to the host or a data transfer from the host to the network.

[0127] If buffer allocation fails (S121: NO), the flow returns to step S118. If buffer allocation is successful (S121: YES), command processing 305 transfers data from the host to the buffer using DMA controller 205 under DMA control 307, or stores data from network controller 203 in the buffer and transfers it from the buffer to the host using DMA controller 205 (S123). In response to the data transfer, host queue control 304 updates the Head of internal receive queue 372 (S124).

[0128] Returning to step S119, if data transfer using a buffer is not required (S119: NO), command processing 305 determines whether data transfer is required (S122). If data transfer without using a buffer is required (S122: YES), DMA controller 205 transfers data between network controller 203 and the host without using a buffer (S123). If data transfer without using a buffer is not required (S122: NO), the next process is performed without transferring data. Host queue control 304 updates the Head of internal receive queue 372 (S124).

[0129] Next, the command processing 305 determines whether it is necessary to transmit a communication packet to the network (S125). If it is not necessary (S125: NO), the flow returns to step S104. If it is necessary (S125: YES), the command processing 305 requests the protocol processing 303, the protocol processing 303 executes the protocol processing to generate a communication packet, and the network controller 203 transmits the communication packet to the network (S126).

[0130] 11 shows an example of a process flow of the initiator network interface 104. For example, the server system 100 that accesses the storage controller 102, or the network interface 104 implemented in the storage controller 102 of the initiator in inter-storage communication operates according to FIG.

[0131] The host loads a host command into the host receive queue 402 (S151) and updates the Tail of the host receive queue 402 (S152). In response to the host updating the information in the host receive queue 402, the network interface 104 also updates the information in its host receive queue 402. The host and the network interface 104 may refer to common information, or may transfer updated information between them.

[0132] Upon receiving the tail update of the host receive queue 402, the command processing 305 of one of the cores uses the DMA controller 205 via the DMA control 307 to transfer the host commands stacked in the host receive queue 402 to the network interface 104 (S153). For example, the cores may periodically obtain a lock to check whether there has been an update to the host receive queue and perform a transfer process if there has been an update, or a core may be provided that is dedicated to checking for updates to the host receive queue and transferring the updates, or multiple host receive queues may be created and a transfer process may be performed if there is an update to the receive queue that the core is responsible for.

[0133] The host queue control 304 loads the host command from the host into the internal receive queue 372 (S154). For example, if it is a new session, the destination internal receive queue 372 may be selected in a round robin manner, or the contents of the host command may be checked to determine which internal receive queue 372 the command should be loaded into, or if there are multiple host receive queues, the command may be loaded into the corresponding internal receive queue 372.

[0134] In response to the loading of the host command into the internal receive queue 372, the host queue control 304 updates the Tail of the internal receive queue 372 (S155), and further updates the Head of the host receive queue 402 (S156). The method of updating the Head of the host receive queue 402 is the same as that described for updating the Tail of the host receive queue 402.

[0135] The command processing 305 acquires the host command from the internal receive queue 372 and executes the processing (S157). If data transfer using a buffer is required (S158: YES), a buffer control step S159 is executed. Details of the buffer control step S159 will be described later. The data transfer is, for example, a data transfer from the host to the network or a data transfer from the host to the network.

[0136] If the buffer allocation fails (S160: NO), the flow returns to step S157. If the buffer allocation is successful (S160: YES), the command processing 305 uses the DMA controller 205 via the DMA control 307 to transfer the data to the buffer (S123). In response to the data transfer, the host queue control 304 updates the Head of the internal receive queue 372 (S163).

[0137] Returning to step S158, if data transfer using a buffer is not required (S119: NO), command processing 305 determines whether data transfer is required (S161). If data transfer without using a buffer is required (S161: YES), DMA controller 205 transfers data between network controller 203 and the host without using a buffer (S123). If data transfer without using a buffer is not required (S61: NO), host queue control 304 updates the Head of internal receive queue 372 (S163).

[0138] Next, the command processing 305 determines whether it is necessary to transmit a communication packet to the network (S164). If it is not necessary (S164: NO), the flow proceeds to step S166. If it is necessary (S164: YES), the command processing 305 requests the protocol processing 303, which executes the protocol processing to generate a communication packet, and the network controller 203 transmits the communication packet to the network (S165).

[0139] Next, the command processing 305 generates a host command for notifying the host of the processing result of the host command from the host (S166). The notification of the processing result is, for example, notifying the host that a communication packet has been transmitted to the network. Furthermore, the command processing 305 loads the generated host command into the internal transmission queue 371 in the network interface 104 (S167). In response to the loading of the host command, the command processing 305 updates the Tail of the internal transmission queue 371 (S168).

[0140] Upon receiving the Tail update of the internal transmit queue 371, the host queue control 304 transfers the host command in the internal transmit queue 371 to the host transmit queue via the DMA control 307 and the DMA controller 205 (S169). In response to the transfer of the host data, the host queue control 304 updates the Tail of the host transmit queue 401 (S170). The network interface 104 and the host may refer to the same information for the Tail and Head information of the host transmit queue 401, or the information held by each may be transferred. Furthermore, the host queue control 304 updates the Head of the internal transmit queue (S171).

[0141] The host acquires the host command from the host transmission queue 401 and updates the Head of the host transmission queue (S172).

[0142] The following describes details of the buffer control process, including the buffer control steps S120 and S159 in Figures 10 and 11. Figure 12 shows a flowchart of an example of the buffer control process. First, the buffer control 306 acquires a buffer allocation request from, for example, the command process 305 (S201).

[0143] The buffer control 306 executes a buffer list determination step (S202). The buffer list determination step determines which of the buffer groups of different sizes to use. The details of the buffer list determination step will be described later.

[0144] Furthermore, the buffer control 306 refers to the buffer list of the selected buffer group and the buffer management table 770, and executes a maximum active number determination step (S203). The maximum active number determination step will be described in detail later. The maximum active number determination step determines whether to execute the current buffer control or to wait. If it is determined in the maximum active number determination step S203 that the buffer control is to be executed, the flow proceeds to step S204.

[0145] The buffer control 306 refers to the buffer list determined in the buffer list determination step S202 (S204), and acquires the buffer entry head address 751 of the buffer list (S205). The buffer control 306 accesses the acquired buffer entry head address, and acquires the buffer entry 703 (S206).

[0146] The buffer control 306 obtains the next buffer entry address 763 from the obtained buffer entry 703 (S207), and further obtains the buffer address 762 (S208).

[0147] Next, the buffer control 306 reflects the next buffer entry address 763 in the buffer entry head address 751 of the buffer list management information 750 (S209). The buffer control 306 holds the used buffer list and buffer entry as task information related to the data transfer (S210). One or more entries of one or more buffer lists may be held as information of a task being executed (not yet completed). The buffer control 306 increments the buffer active number 752 of the buffer list management information 750 (S211).

[0148] A data transfer using a buffer is executed (S212), and when the data transfer is completed (S213), the buffer control 306 identifies the buffer list and buffer entry to which the data transfer corresponds, based on the task information related to the data transfer held in step 210. The buffer control 306 obtains the buffer entry head address 751 of that buffer list (S215).

[0149] The buffer control 306 reflects the acquired buffer entry head address 751 to the next buffer entry address 763 of the buffer entry specified from the task information (S216). The buffer control 306 sets the address 761 of the buffer entry specified from the task information to the buffer entry head address 751 (S217). Finally, the buffer control 306 decrements the buffer active number 752 of the buffer list.

[0150] According to the above operation, the buffer entry for which the data transfer was completed immediately before returns to the head of the buffer list. The next data transfer uses the buffer entry at the head of the buffer list. The cache area stores the data of the associated buffer area. Therefore, the same buffer is reused, and if the data of that buffer was cached, a cache hit occurs. This can reduce the number of memory accesses. Note that the released buffer may be returned to a position different from the head in the buffer list. For example, it may be returned before the preset order. In the above example, the head is the preset order.

[0151] Next, the process of the buffer list determination step S202 will be described. FIG. 13 shows a flowchart of an example of buffer list determination processing. In this example, it is assumed that three different sizes of buffers, X, Y, and Z, are managed by corresponding buffer lists. The size relationship is as follows: X < Y < Z. For example, X = 8KB, Y = 32KB, and Z = 256KB may be used. Note that the number of types of buffer sizes is arbitrary, and each size is also arbitrary.

[0152] The buffer control 306 acquires the data size to be transferred from the host command (S251). If the data size is X or less (S252: YES), the buffer control 306 selects the buffer list of size X (S253). If the data size is greater than X and Y or less (S252: NO, S254: YES), the buffer control 306 selects the buffer list of size Y (S255).

[0153] If the data size is greater than Y and equal to or less than Z (S254: NO, S256: YES), the buffer control 306 selects a buffer list of size Z (S253). If the data size is greater than Z (S256: NO), the buffer control 306 reserves a long-size buffer greater than Z in the memory 208 (S258).

[0154] As described above, by sequentially checking from the small size buffer list, a buffer with the smallest buffer size among those that are equal to or larger than the transfer data size is selected to store the transfer data. This allows the buffer area to be used efficiently. The reason for reserving buffer area for long size data on the fly is that with long data sizes, the data transfer time by DMA205 becomes longer, and the processor load during data transfer is low, so even if the buffer area is reserved each time, the processor load is low. Also, long size data easily overflows from the cache and memory access occurs, so list management is not required.

[0155] Next, an example of performing buffer list determination using a method different from that used in buffer list determination step S202 will be described. In the example described with reference to Figures 12 and 13, transfer data is stored in one buffer. In the example described below, large transfer data is distributed and stored in multiple buffers. This allows transfer data larger than the buffer size managed in the buffer list to be stored in the buffer managed in the buffer list.

[0156] Fig. 14 is a flowchart of an example of buffer control processing. Differences from the flowchart shown in Fig. 12 will be mainly described. The buffer control 306 executes a buffer list determination step S271. The buffer list determination step S271 differs from the buffer list determination step S202 in Fig. 12. After step S211, the buffer control 306 executes step S273, and after step S218, executes step S275.

[0157] First, the buffer list determination step S271 will be described. Fig. 15 shows a flowchart of a processing example in the buffer list determination step S271. The buffer control 306 acquires the remaining data size of the data to be transferred from the host command (S301). If the remaining data size is equal to or less than X (S302: YES), the buffer control 306 selects a buffer list of size X, and further reduces the remaining data size by X (S303).

[0158] If the remaining data size is greater than X and equal to or less than Y (S302: NO, S304: YES), the buffer control 306 selects a buffer list of size Y and further reduces the remaining data size by Y (S305). If the remaining data size is greater than Y and equal to or less than Z (S304: NO, S306: YES), the buffer control 306 selects a buffer list of size Z and reduces the remaining data size by Z (S307).

[0159] If the remaining data size is greater than Z (S306: NO), or after any of steps S303, S306, or S307 has been executed, the buffer control 306 determines whether the remaining data size exists (S308). If the total size of the allocated buffers is equal to or greater than the transfer data size, the remaining data size is 0. If the remaining data size exists (S308: YES), the flow returns to step S302. If the remaining data size does not exist (S308: NO), the buffer list determination process ends.

[0160] As described with reference to Fig. 15, large-sized transfer data is divided into a plurality of partial data, and a buffer of a size corresponding to each of the partial data is assigned to each of the partial data. Returning to Fig. 14, the buffer control 306 executes steps S204 to S211 for each partial data. Step S273 determines whether steps S204 to S211 have been executed for all partial data. After the data transfer is completed, the buffer control 306 executes steps S215 to S218 for each partial data. Step S275 determines whether steps S215 to S218 have been executed for all partial data.

[0161] The process of the maximum active number determination step S203 in Fig. 12 and Fig. 14 will be described below. Fig. 16 shows a flowchart of an example of the maximum active number determination process. In the following, it is assumed that each core 241 is assigned an individual cache threshold and an individual buffer list group.

[0162] The buffer control 306 acquires the buffer list determination result (S401) Next, the buffer control 306 acquires the buffer active number 752 from the buffer list management information 750 of each buffer list (S402).

[0163] The buffer control 306 calculates the total number of the buffer active number 752 and the number of buffers scheduled to be used (scheduled active buffer number) for each buffer list scheduled to be used indicated by the buffer list determination result, and compares this total number with the maximum active number 774 (S403). If the scheduled active buffer number of any buffer list exceeds the maximum active number 774 (S403: YES), the buffer control 306 determines to suspend processing and wait for a predetermined time (S404). After this, the flow returns to the buffer list determination steps S202 and S271.

[0164] If the planned active buffer count for each buffer list is equal to or less than the corresponding maximum active count 774 (S403: NO), the buffer control 306 calculates the total amount of active buffers, including the buffer amount planned for current use (S405). Specifically, the buffer control 306 calculates the sum of products of the size of the buffer list assigned to the core and the active buffer count ((X size x X active count) + (Y size x Y active count) + (Z size x Z active count)). The buffer control 306 adds the buffer amount planned for current use to this sum of products.

[0165] Next, the buffer control 306 compares the total active buffer amount with the cache threshold assigned to the core (S406). The cache threshold is a value equal to or less than the cache capacity assigned to the core. If the total active buffer amount is equal to or less than the cache threshold (S406: NO), this maximum active number determination process ends. If the total active buffer amount is greater than the cache threshold (S406: YES), the buffer control 306 determines to suspend processing and wait (S404). After this, a predetermined time has passed, and the buffer control process is started again.

[0166] As described above, the network interface 104 controls the command processing that can be started based on the difference between the cache capacity and the total amount of active buffer usage. To prevent the cache from overflowing, the number of data transfers processed simultaneously is limited, and the number of buffers in use is also limited. This makes it possible to prevent performance degradation due to memory bandwidth pressure by reducing memory accesses through cache hits.

[0167] In the above example, each core is assigned its own set of buffer lists and cache thresholds. A separate cache area may be assigned to each core, or one cache area may be shared, i.e., used, by multiple cores.

[0168] In another example, the cache region and all buffer lists may be shared by all cores, and a cache threshold may be defined for the shared cache region, i.e., the cache threshold may be shared by the cores. In this example configuration, each core may perform a max actives determination process as described with reference to FIG.

[0169] In another example, a cache area may be shared by all cores, a cache threshold may be defined for the shared cache area, and a buffer list group may be assigned to each core. In this configuration example, each core calculates the total active buffer size by referring to the buffer list group assigned to itself as well as the buffer list group assigned to other cores.

[0170] In another example configuration, the cache threshold may be set individually for each core, and the buffer lists may be shared by the cores. Each core may store information about the buffers it is using as task information, and may compare the total amount of buffers it is using with its individual cache threshold.

[0171] In another example configuration, the cores may be grouped into groups, and a cache area may be assigned to each group. For example, a cache threshold and buffer lists may be assigned for each group. Each core may use the buffer lists and cache threshold for the same group to perform the max actives determination process as described with reference to FIG.

[0172] 16, the permission of buffer allocation is determined based on the maximum active number set for each buffer list (S403). In another example, this step S403 may be omitted, and the permission of buffer allocation may be determined based only on the cache usage.

[0173] As described above, the cores may be divided into multiple groups, and different cache areas may be assigned to each group. In this configuration, the network interface 104 may perform performance balancing among multiple network ports, thereby improving the performance of the network interface 104.

[0174] For example, assume that there are eight cores 0 to 7, with two cores constituting each group. Here, assume that cores 0 and 1 constitute one group, cores 2 and 3 constitute one group, cores 4 and 5 constitute one group, and cores 6 and 7 constitute one group. Assume that a cache area and a buffer list group are assigned to each group. Assume that the network interface 104 has two ports, port 0 and port 1.

[0175] The network interface 104 assigns the cores in each group to different ports. For example, core 0, core 2, core 4, and core 6 are assigned to port 0, and core 1, core 3, core 5, and core 7 are assigned to port 1. For example, a cache threshold that is half the cache capacity is assigned to each core in each group. This makes it possible to equalize the cache amounts used by port 0 and port 1, thereby balancing performance. Note that the cache thresholds of the cores in each group do not need to be the same.

[0176] Alternatively, the network interface 104 may assign the cores of a group to the same port. For example, core 0, core 1, core 2, and core 3 are assigned to port 0, and core 4, core 5, core 6, and core 7 are assigned to port 1. In each group, a cache threshold may be set for each port, or a cache threshold set for a cache area may be used for both of the two cores.

[0177] Another example of the maximum active number determination process will be described below. FIG. 17 shows a flowchart of another example of the maximum active number determination process. The following mainly describes the differences from the processing example shown in FIG. 16. This example of the maximum active number determination process is executed together with the buffer list determination process described with reference to FIG. 13. In this processing example, when a new buffer area is secured (step S258 in FIG. 13), data is transferred without determining the remaining cache capacity. If the transfer data size is large, there is a high possibility that it will overflow from the cache. This example can improve the efficiency of the maximum active number determination process.

[0178] 17, after acquiring the buffer list determination result (S401), the buffer control 306 determines whether a new buffer area has been allocated (S411). If a new buffer area has not been allocated (S411: NO), the buffer control 306 executes steps S402 and onward. These steps are as described with reference to FIG 16. If a new buffer area has been allocated (S411: YES), the maximum active number determination process is completed.

[0179] Fig. 18 shows a flowchart of another example of maximum active number determination processing. The following mainly describes the differences from the processing example shown in Fig. 16. In this example, when a data transfer request exceeds the maximum buffer size, data is transferred without determining the remaining cache capacity. This makes the processing more efficient.

[0180] 18, after the buffer control 306 acquires the buffer list determination result (S401), it determines whether the transfer data size exceeds the maximum buffer size (S421). If it does not exceed the maximum buffer size (S421: YES), the buffer control 306 executes step S402 and subsequent steps. These steps are as described with reference to FIG 16. If the transfer data size exceeds the maximum buffer size (S421: NO), the maximum active number determination process is completed.

[0181] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0182] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0183] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other.

[0184] [Additional Notes] Several aspects of the disclosure are described below. (1) A network interface, A processor; Memory, a cache that is accessed by the processor before the memory; The processor, allocating a plurality of buffers in said memory for storing transfer data; Manage an allocation order of free buffers among the plurality of buffers; The buffer released after the data transfer is returned to the top position of the allocation order as a free buffer; A network interface used for transferring data from the free buffer at the head position. (2) A network interface according to (1), The processor reserves a plurality of buffer groups in the memory; Each of the plurality of buffer groups is composed of a plurality of buffers of the same size, The plurality of buffer groups have different buffer sizes, A network interface, wherein when a transfer data size is equal to or smaller than a maximum buffer size of the different buffer sizes, the processor selects a buffer group having a minimum buffer size among buffer sizes equal to or larger than the transfer data size for storing the transfer data. (3) A network interface according to (1), The processor determines whether or not to allocate a buffer for new transfer data based on the amount of the buffer in use. (4) A network interface according to (2), If the transfer data size is larger than the maximum buffer size, the processor secures a new buffer area equal to or larger than the transfer data size. (5) A network interface according to (2), When the transfer data size is larger than the maximum buffer size, the processor divides the transfer data and allocates a buffer selected from the group of buffers. (6) A network interface according to (2), A network interface, wherein a maximum number of buffers that can be used simultaneously is set in each of the plurality of buffer groups. (7) A network interface according to (2), The processor, If the transfer data size is equal to or smaller than the maximum buffer size, determining whether or not to allocate a buffer for new transfer data based on the amount of buffer being used; If the transfer data size is larger than the maximum buffer size, the network interface skips the determination and reserves a buffer for storing the transfer data. (8) A network interface according to (1), The processor determines whether or not to allocate a buffer for new transfer data based on the amount of buffer in use; A network interface, wherein a maximum number of buffers that can be used simultaneously is set among the multiple buffers. (9) A method for controlling a buffer in a network interface, comprising the steps of: the network interface includes a processor, a memory, and a cache that is accessed by the processor prior to accessing the memory; The buffer control method includes the processor: allocating a plurality of buffers in said memory for storing transfer data; Manage an allocation order of free buffers among the plurality of buffers; The buffer released after the data transfer is returned to the top position of the allocation order as a free buffer; A buffer control method used for data transfer from the empty buffer at the head position. (10) A buffer control method according to (9), The processor reserves a plurality of buffer groups in the memory; Each of the plurality of buffer groups is composed of a plurality of buffers of the same size, The plurality of buffer groups have different buffer sizes, The buffer control method includes, when a transfer data size is equal to or smaller than a maximum buffer size of the different buffer sizes, selecting a group of buffers having a minimum buffer size among buffer sizes equal to or larger than the transfer data size to store the transfer data. (11) A buffer control method according to (9), The buffer control method includes determining whether or not to allocate a buffer for new transfer data based on an amount of buffer in use. (12) A buffer control method according to (10), The buffer control method further comprises the steps of: when the transfer data size is larger than the maximum buffer size, the processor secures a new buffer area equal to or larger than the transfer data size. (13) A buffer control method according to (10), The buffer control method, wherein, when the transfer data size is larger than the maximum buffer size, the processor divides the transfer data and assigns a buffer selected from the group of multiple buffers. (14) A buffer control method according to (10), A buffer control method, wherein a maximum number of buffers that can be used simultaneously is set in each of the plurality of buffer groups. (15) A buffer control method according to (10), When the transfer data size is equal to or smaller than the maximum buffer size, the processor determines whether or not to allocate a buffer for new transfer data based on an amount of buffer space in use; The buffer control method, wherein, when the transfer data size is larger than the maximum buffer size, the processor skips the determination and reserves a buffer for storing the transfer data. (16) A buffer control method according to (9), The processor determines whether or not to allocate a buffer for new transfer data based on the amount of buffer in use; A buffer control method, wherein a maximum number of buffers that can be used simultaneously is set among the multiple buffers. [Explanation of symbols]

[0185] 100 server system, 101 storage device, 102 storage controller, 104 network interface, 201 host bus, 200 network path, 202 data processing unit, 208 memory, 208 memory controller, 203 network controller, 204 processor, 205 DMA controller, 206 cache, 241 processor core, 301 network controller driver, 303 protocol processing, 305 command processing, 307 DMA control, 304 host queue control, 306 buffer control, 307 DMA control, 371 internal transmit queue, 372 internal receive queue, 401 host transmit queue, 402 host receive queue, 400 host, 450 remote site, 700-702 buffer list, 703 buffer entry, 750 buffer list management information, 770 buffer management table

Claims

1. A network interface, A processor; Memory, a cache that is accessed by the processor before the memory; The processor, allocating a plurality of buffers in said memory for storing transfer data; Determine whether to allocate a buffer for new transfer data based on the amount of buffer in use; A network interface, wherein a maximum number of buffers that can be used simultaneously is set among the multiple buffers.

2. 2. The network interface of claim 1, The processor returns the buffer released after the data transfer to a top position in the allocation order of free buffers of the plurality of buffers as a free buffer.

3. 2. The network interface of claim 1, The processor reserves a plurality of buffer groups in the memory; Each of the plurality of buffer groups is composed of a plurality of buffers of the same size, The plurality of buffer groups have different buffer sizes, A network interface, wherein when a transfer data size of first transfer data is equal to or smaller than a maximum buffer size of the different buffer sizes, the processor selects a buffer group having a minimum buffer size among buffer sizes equal to or larger than the transfer data size to store the first transfer data.

4. 2. The network interface of claim 1, The processor determines whether or not to allocate a buffer for new transfer data based on the amount of the buffer in use.

5. 4. A network interface according to claim 3, If the transfer data size is larger than the maximum buffer size, the processor secures a new buffer area equal to or larger than the transfer data size.

6. 4. A network interface according to claim 3, If the transfer data size is larger than the maximum buffer size, the processor divides the first transfer data and allocates a buffer selected from the group of buffers.

7. 4. A network interface according to claim 3, A network interface, wherein a maximum number of buffers that can be used simultaneously is set in each of the plurality of buffer groups.

8. 4. A network interface according to claim 3, The processor, If the transfer data size is equal to or smaller than the maximum buffer size, determining whether or not to allocate a buffer for new transfer data based on the amount of buffer being used; If the transfer data size is larger than the maximum buffer size, the network interface skips the determination and reserves a buffer for storing the transfer data.

9. 2. The network interface of claim 1, The processor Manage an allocation order of free buffers among the plurality of buffers; The network interface returns the buffer released after the data transfer to a position before a predetermined position in the allocation order as a free buffer.

10. A method for controlling a buffer in a network interface, comprising the steps of: the network interface includes a processor, a memory, and a cache that is accessed by the processor prior to accessing the memory; The buffer control method includes the processor: allocating a plurality of buffers in said memory for storing transfer data; Based on the amount of buffer space in use, a determination is made as to whether or not a buffer space should be allocated for new data to be transferred; A buffer control method, wherein a maximum number of buffers that can be used simultaneously is set among the multiple buffers.

Images