A method and system for resuming interrupted data transfer on a smart network card

Abstract

This application provides a method and system for resuming interrupted data transmission on a smart network interface card (NIC). This application is server-independent, requires no modification to the client's operating system, and can handle failures under any circumstances, including FPGA transmission failures, on-card storage software failures, operating system failures, and remote cloud storage transmission failures. A hash table is used to cache requests, which can handle out-of-order responses from storage. The hash table only caches the request sequence number, not the request content or message data, reducing FPGA resource consumption and saving costs.

Description

Technical Field

[0001] This application relates to the field of storage acceleration technology for smart network interface cards (NICs), specifically to a storage breakpoint resume method and system for smart NICs. Background Technology

[0002] Smart NICs, also known as intelligent network adapters, not only perform the network transmission functions of standard NICs but also provide a built-in programmable and configurable hardware acceleration engine. This enhances application performance and significantly reduces CPU consumption during communication, freeing up more CPU resources for applications. Smart NICs can achieve accelerated processing through CPUs, FPGAs, or ASICs. The smart NICs referred to in this invention are specifically FPGA-based programmable smart NICs.

[0003] In the process of digital transformation and cloud adoption by the internet industry and traditional enterprises, more and more companies are placing their storage systems in the cloud due to the advantages of cloud storage, such as scalability, flexibility, multi-tenancy, terminal-free data migration, and lower disaster recovery costs. In data center scenarios, smart network interface cards (NICs) are commonly used to provide network and storage acceleration for servers. Storage acceleration provides servers with transparent cloud storage services, allowing them to use storage like local hard drives without network processing overhead, while still enjoying the flexibility and reliability of cloud storage. Smart NICs present standard virtio-blk storage devices to the server, enabling local storage access; simultaneously, the smart NIC sends the received storage data to the cloud via the network, achieving cloud storage. While this application model seamlessly presents cloud storage to the server, it also lengthens the processing path for storage services, introducing more instability. The application of smart NICs for storage acceleration also brings problems such as excessively long processing paths and increased points of failure. For example, lost storage packets can cause the server's storage driver to freeze and become unrecoverable.

[0004] (1) The server sees a standard virtio-blk disk. Current operating systems like Linux have a relatively simple access logic for local disks, assuming that every storage request can be processed and will definitely receive a response from the "hardware". The operating system does not have a timeout retransmission mechanism and will wait indefinitely for a response from the "hardware".

[0005] (2) The smart network card uses the FPGA to send the storage data sent from the server to the CPU on the card for processing, and finally transmits it to the cloud storage via the network. This process involves several stages: FPGA transmission, CPU software transmission, and network transmission. If any of these stages fail, for example, if the CPU's operating system fails and restarts, the storage request will be lost, causing the server to freeze because it cannot wait for a storage response. Summary of the Invention

[0006] To address the aforementioned technical problems, this application implements a method and system for resuming interrupted data transfers on a smart network interface card (NIC), enabling the rapid recovery of storage services. The technical solution adopted in this application is as follows:

[0007] A method for resuming interrupted data transfers on a smart network interface card (NIC), comprising the following steps:

[0008] Step 1: The DMA engine of the smart network card FPGA obtains a data transmission request from the available queue, including the location, length, and disk address of the data to be transmitted; the DMA engine initiates data transmission according to the data transmission request, and caches the sequence number of the data transmission request in a hash table; each data transmission request sequence number carries a timestamp.

[0009] Step 2: The DMA engine in the smart network card FPGA attaches metadata to the data transmission request and sends it to the smart network card CPU. The metadata includes the transmission sequence number.

[0010] Step 3: The smart network card CPU receives the data transmission request and performs transmission fault detection. If no transmission fault occurs, proceed to step 4; if a transmission fault occurs, perform fault recovery, return and proceed to step 2.

[0011] Step 4: The storage software of the smart network card CPU sends a data reception success feedback to the smart network card FPGA. The data reception success feedback carries the sequence number of the data transmission request corresponding to the successfully received data.

[0012] Furthermore, prior to step 1, the method also includes: when a user has data to send, the Virtio front-end driver located on the server side puts the data transmission request into the virtio avail queue.

[0013] Furthermore, step 4 also includes: after the smart network card FPGA receives the data reception feedback, it deletes the sequence number of the data transmission request temporarily stored in the hash table, indicating that the data transmission request has been successfully completed.

[0014] Furthermore, in step 3, transmission fault detection is performed, including: the smart network card CPU detects whether the transmission sequence number is continuous to determine whether a transmission fault has occurred in the smart network card FPGA, thereby realizing early detection of data loss; if the transmission sequence number is not continuous, a transmission fault has occurred, and fault recovery is performed.

[0015] Furthermore, if the storage software of the smart network card CPU fails and restarts, or if the storage software restarts due to a failure in the operating system, the storage software will perform fault recovery upon restarting.

[0016] Furthermore, the smart network interface card (FPGA) checks the timestamp carried by the sequence number of the data transmission request in the hash table. If the data transmission request times out, the smart network interface card (FPGA) notifies the decision software to determine whether to initiate fault recovery.

[0017] Furthermore, the fault recovery process includes: the smart network card FPGA obtaining all transmission sequence numbers from the hash table, obtaining transmission information again from the server's virtio-blk available queue, and initiating DMA transmission.

[0018] A storage breakpoint resume system on a smart network card, the system being used to implement the above method, the storage breakpoint resume system comprising a server, a smart network card CPU, and a smart network card FPGA.

[0019] Furthermore, in the smart network interface card (FPGA), a hash table is added for each virtio-blk transmission queue. The hash table is used to store the sequence number of the data transmission request and supports adding, deleting, and querying by key.

[0020] Furthermore, the hash table only caches the sequence number of the data request, and does not cache the content of the data transmission request.

[0021] Through the embodiments of this application, the following technical effects can be achieved: In a public cloud, the server runs the client's operating system and services; this invention is unaware of the server and does not require modification of the client's operating system. This application can handle failures under any circumstances, including FPGA transmission failures, on-card storage software failures, operating system failures, and remote cloud storage transmission failures. Using a hash table to cache requests can handle out-of-order responses from storage. The hash table only caches the request sequence number, not the request content or message data, reducing FPGA resource consumption and saving costs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the structure of a storage breakpoint resume system;

[0024] Figure 2 This is a schematic diagram of another component structure of a storage breakpoint resume system. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] like Figure 1 As shown, the storage breakpoint resume system includes a server, a smart network card CPU, and a smart network card FPGA.

[0027] In the smart network interface card (FPGA), a hash table is added for each virtio-blk transmission queue. The hash table is used to store the sequence number of the data transmission request and supports adding, deleting and querying by key.

[0028] When the smart network interface card (FPGA) receives a data transmission request, it inserts the sequence number of the data transmission request into the hash table for caching. When the corresponding data response for the data transmission request is received, the sequence number of the data transmission request is deleted from the hash table. When a transmission failure occurs, the smart network interface card (FPGA) queries the hash table to obtain the data transmission request lost during the transmission failure. The smart network interface card (FPGA) uses the cached data transmission request to retrieve the data from the server and initiate transmission again.

[0029] The hash table only caches the sequence number of the data request, not the content of the data transmission request. Because the content of the data transmission request persists in the server's memory, using the server as a cache minimizes the overhead of the smart network interface card (FPGA). Furthermore, fault recovery is not performance-sensitive, and retrieving data from the server's memory again is acceptable.

[0030] The method for resuming interrupted downloads includes the following steps:

[0031] Step 1: The DMA engine of the smart network card FPGA obtains a data transmission request from the available queue, including the location, length, and disk address of the data to be transmitted; the DMA engine initiates data transmission according to the data transmission request, and caches the sequence number of the data transmission request in a hash table; each data transmission request sequence number carries a timestamp.

[0032] Before step 1, the method also includes: when a user has data to send, the Virtio front-end driver located on the server side puts the data transmission request into the virtio avail queue;

[0033] Step 2: The DMA engine in the smart network card FPGA attaches metadata to the data transmission request and sends it to the smart network card CPU. The metadata includes the transmission sequence number.

[0034] Step 3: The smart network card CPU receives the data transmission request and performs transmission fault detection. If no transmission fault occurs, proceed to step 4; if a transmission fault occurs, perform fault recovery, return and proceed to step 2.

[0035] Step 4: The storage software of the smart network card CPU sends a data reception success feedback to the smart network card FPGA. The data reception success feedback carries the sequence number of the data transmission request corresponding to the successfully received data.

[0036] Step 4 also includes: after the smart network card FPGA receives the data reception feedback, it deletes the sequence number of the data transmission request temporarily stored in the hash table, indicating that the data transmission request has been successfully completed;

[0037] In step 3, transmission fault detection is performed, including:

[0038] The CPU of the smart network card detects whether the transmission sequence number is continuous to determine whether a transmission fault has occurred in the smart network card FPGA, thus realizing early detection of data loss; if the transmission sequence number is not continuous, a transmission fault has occurred, and fault recovery is performed.

[0039] If the storage software of the smart network card CPU fails and restarts, or if the storage software restarts due to a failure in the operating system, the storage software will perform fault recovery upon restarting.

[0040] The smart network card FPGA checks the timestamp carried by the sequence number of the data transmission request in the hash table. If the data transmission request times out, the smart network card FPGA notifies the decision software to determine whether to initiate fault recovery.

[0041] The fault recovery process includes: the smart network card FPGA obtaining all transmission sequence numbers from the hash table, obtaining transmission information again from the server's virtio-blk available queue, and initiating DMA transmission.

[0042] The technical solution of this application will be further explained below in conjunction with specific application scenarios.

[0043] Application Scenario 1: Flexible Bare Metal Scenarios

[0044] In public cloud elastic bare metal products, smart network interface cards (NICs) present PCIe virtio-blk devices to the server. The CPU on the smart NIC runs SPDK software to enable remote cloud storage access.

[0045] The FPGA of the smart network card implements the functions of this application. When the CPU's SPDK software detects that the sequence number of the sequence message transmitted from the FPGA is not continuous, it actively triggers the FPGA to initiate a retransmission.

[0046] An SPDK software malfunction caused the program to crash. Under the monitoring of the operating system, the SPDK software was restarted. When the SPDK ran, it first requested the FPGA to initiate a retransmission in order to recover the data lost during the software crash.

[0047] When the Linux operating system running on the CPU malfunctions and the operating system restarts, the SPDK software will automatically start after the restart. SPDK will first request the FPGA to initiate a retransmission to recover the data lost during the software crash.

[0048] Application Scenario 2: Virtualization Scenarios

[0049] In public cloud virtualization scenarios, smart network interface cards (NICs) present PCIe virtio-blk devices to the server and, through PCIe passthrough technology, directly pass the virtio-blk data to the virtual machine. The virtual machine's data is then sent directly to the FPGA, bypassing the host's software and achieving optimal I / O performance.

[0050] Virtualization scenarios require support for live migration technology. During live migration, storage requests sent to the FPGA may not have been completed yet, leading to the loss of some in-flight data. Therefore, the traditional solution is to pause the virtual machine's operation, waiting for the requested data to be received before continuing the live migration. This results in excessively long pause times for the virtual machine, significantly impacting user services.

[0051] The technical solution of this application allows for immediate hot migration without waiting for data to be ready. After the virtual machine is migrated, an FPGA breakpoint resume is initiated first, and requests that have not received a response are re-initiated. This achieves the purpose of accelerating I / O with the smart network card and enables rapid migration of virtual machines.

[0052] In summary, this application provides a general fault recovery method for various potential faults of smart network interface cards (NICs), enabling storage interruption resume functionality and rapid recovery of storage services. Furthermore, this application achieves server-insensitive fault recovery. In scenarios similar to public clouds, servers typically run customer systems, making server-insensitive fault recovery for smart NICs truly meaningful.

[0053] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0054] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0055] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0056] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0057] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0058] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0059] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0060] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for resuming interrupted data transfer on a smart network card, characterized in that, The method includes the following steps: Step 1: The DMA engine of the smart network card FPGA obtains a data transmission request from the available queue, including the location, length, and disk address of the data to be transmitted; the DMA engine initiates data transmission according to the data transmission request, and caches the sequence number of the data transmission request in a hash table; each data transmission request sequence number carries a timestamp. Step 2: The DMA engine in the smart network card FPGA attaches metadata to the data transmission request and sends it to the smart network card CPU. The metadata includes the transmission sequence number. Step 3: The smart network card CPU receives the data transmission request and performs transmission fault detection. If no transmission fault occurs, proceed to step 4. If a transmission failure occurs, perform fault recovery, return, and proceed to step 2; Step 4: The storage software of the smart network card CPU sends a data reception success feedback to the smart network card FPGA. The data reception success feedback carries the sequence number of the data transmission request corresponding to the successfully received data. In step 3, transmission fault detection is performed, including: the smart network card CPU detects whether the transmission sequence number is continuous to determine whether a transmission fault has occurred in the smart network card FPGA, thereby realizing early detection of data loss; if the transmission sequence number is not continuous, a transmission fault has occurred, and fault recovery is performed. If the storage software of the smart network card CPU fails and restarts, or if the storage software restarts due to a failure of the operating system, the storage software will perform fault recovery upon restarting. The smart network card FPGA checks the timestamp carried by the sequence number of the data transmission request in the hash table. If the data transmission request times out, the smart network card FPGA notifies the decision software to determine whether to initiate fault recovery. The fault recovery process includes: the smart network card FPGA obtaining all transmission sequence numbers from the hash table, obtaining transmission information again from the server's virtio-blk available queue, and initiating DMA transmission.

2. The method according to claim 1, characterized in that, Before step 1, the method also includes: when a user has data to send, the Virtio front-end driver located on the server side puts the data transmission request into the virtio avail queue.

3. The method according to claim 1, characterized in that, Step 4 also includes: after the smart network card FPGA receives the data reception feedback, it deletes the sequence number of the data transmission request temporarily stored in the hash table, indicating that the data transmission request has been successfully completed.

4. A storage breakpoint resume system on a smart network card, the system being used to implement the method as described in any one of claims 1 to 3, characterized in that, The storage breakpoint resume system includes a server, a smart network card CPU, and a smart network card FPGA.

5. The system according to claim 4, characterized in that, In the smart network interface card (FPGA), a hash table is added for each virtio-blk transmission queue. The hash table is used to store the sequence number of the data transmission request and supports adding, deleting and querying by key.

6. The system according to claim 5, characterized in that, The hash table only caches the sequence number of the data request, not the content of the data transmission request.

Images