Multi-mode file transfer method based on full user-mode QUIC protocol

By using the full-user-space QUIC protocol, DPDK and libevent are used to process data packets in user space. Combined with breakpoint retransmission and small file aggregation, the inefficiency of traditional file transfer protocols is solved, and file transfer efficiency is improved.

CN118540322BActive Publication Date: 2024-11-05ARTIFICIAL INTELLIGENCE RES INST OF HEFEI COMPREHENSIVE NAT SCI CENT (ANHUI ARTIFICIAL INTELLIGENCE LAB)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411018405.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-05
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

Traditional file transfer protocols, such as TCP based on the operating system kernel, suffer from problems such as poor congestion control algorithms, reliance on operating system updates for version upgrades, and low packet processing efficiency, making them unable to meet the high-efficiency file transfer requirements of modern network environments.

Method used

It adopts the full user-space QUIC protocol, utilizes the DPDK framework and libevent event notification library to migrate data packet processing to user space, avoids kernel dependencies, and optimizes transmission efficiency through breakpoint retransmission strategy and small file aggregation.

Benefits of technology

It significantly improves the speed of large file transfer and the efficiency of small file transfer, reduces system calls and data copying, improves network bandwidth utilization, and reduces retransmission overhead and connection switching costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118540322B_ABST
    Figure CN118540322B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-mode file transmission method based on the full user state QUIC protocol, relates to the technical field of digital information transmission, comprises a client and a server, the client comprises an application, a QUIC protocol, a libevent event notification library, a user state protocol stack and a DPDK, and the multi-mode file transmission method comprises the following steps: the client requests the server for file information to be transmitted, the network card receives the data packet sent by the server, the network card and the protocol stack transmit the data packet through the DPDK, the protocol stack and the application layer transmit the data packet relying on libevent, and then the application layer sends the data packet to the application of the client, so that the client obtains the file information from the server; the multi-mode file transmission method implements the QUIC protocol in the full user state, and improves the data transmission efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of digital information transmission technology, and in particular to a multi-mode file transmission method based on the full user-mode QUIC protocol. Background Art

[0002] In recent years, with the development of network technology, the data generated by the operation of various industries has exploded, the capacity of single files has also increased rapidly, and the pressure of file transfer has increased sharply. As a basic function of the network, file transfer is crucial to the user's network experience and work efficiency. At the same time, file transfer is also a key indicator for testing the performance of transmission protocols and network quality. With the rapid growth of file capacity and network bandwidth, traditional file transfer protocols are also facing more and more problems, and it is urgent to improve the efficiency of file transfer.

[0003] The traditional file transfer method mainly uses the File Transfer Protocol (FTP). Users connect to the remote FTP server by using a client that supports FTP, and issue commands to the server to obtain the corresponding files. During the interaction between the client and the server, the transmission of files and commands depends on the Transmission Control Protocol (TCP) of the operating system kernel. However, with the development of network technology and the rapid growth of network bandwidth, FTP based on the operating system kernel has exposed the following key problems and cannot meet user needs.

[0004] 1) The TCP protocol of the operating system kernel protocol stack uses very conservative congestion control algorithms, such as Reno, SACK, Cubic, etc. These heuristic congestion control algorithms determine network congestion by detecting packet loss. Once packet loss occurs, the congestion window will be greatly reduced, reducing the transmission rate. However, in the current network environment, the link bandwidth is high and the proportion of wireless links is increasing. Packet loss is not necessarily caused by network congestion. Therefore, the traditional TCP congestion algorithm based on packet loss performs poorly.

[0005] 2) The version upgrade of the operating system kernel protocol stack depends on the update of the operating system, which makes it difficult to iterate quickly. In particular, the file transfer protocol is based on the client-server architecture. Upgrading only the server system cannot meet the usage requirements, and the client system needs to be upgraded synchronously. This constraint makes it difficult to quickly deploy the newly developed and more excellent technology into practical applications.

[0006] 3) For large file transfer scenarios, the traditional data packet receiving and sending architecture based on the operating system kernel will become a performance bottleneck. First, after receiving the data packet, the network card will send a hard interrupt to the CPU. The CPU overhead is very large to handle frequent interrupts. Second, the data packet needs to be parsed in the kernel protocol stack and then copied to the application layer of the user space. The copying process is very time-consuming. Third, the multi-core server processes data tasks across cores, which easily leads to cache misses and reduces the processing speed. In the large file transfer scenario, a large number of giant long data packets will appear in a short period of time. The impact of these three disadvantages is more obvious, seriously reducing the transmission efficiency. Summary of the invention

[0007] Based on the technical problems existing in the background technology, the present invention proposes a multi-mode file transfer method based on the full-user state QUIC protocol, implements the QUIC protocol in the full user state, and improves data transmission efficiency.

[0008] The multi-mode file transmission method based on the full user-mode QUIC protocol proposed in the present invention includes a client and a server, wherein the client includes an application, a QUIC protocol, a libevent event notification library, a user-mode protocol stack and DPDK, and the multi-mode file transmission method includes the following steps:

[0009] S1, the client requests the server for information about the file to be transferred;

[0010] S2. The client's network card obtains the data packet sent by the server and stores it in the local buffer. The client's DPDK obtains the data packet from the client's network card by polling, and encapsulates the data packet into the rte_mbuf structure of the structure list of the DPDK data packet storage, and converts the structure list of the DPDK data packet storage into the structure list of the user mode protocol stack;

[0011] S3. The QUIC protocol of the client's application layer first initializes the event and sets the callback function of the event, adds the initialized event to the client's libevent event notification library, completes the event registration, and libevent mounts the initialized event in the libevent event notification library to the waiting list, enters an infinite loop, and waits for the event to be triggered. The data packet parsed from the structure list of the user-mode protocol stack triggers the event in the waiting list. The triggered event is separated from the waiting list and becomes a ready event. Libevent calls the callback function of the ready event to read the data packet from the user-mode protocol stack and sends the read data packet to the client's application to enable the client to obtain file information from the server.

[0012] Furthermore, the rte_mbuf structure header stores flag information, which includes an address, a packet length, and a data length. Head and tail space is reserved in the payload part for user customization, and other space is used to store data packets. For giant long data packets, multiple rte_mbuf structures are linked for storage.

[0013] Furthermore, the user-mode protocol stack uses the mbuf structure to store data packets. The mbuf structure consists of a header and a data area. The header occupies 20 bytes and contains a linked list pointer and data length. The data area consists of 108 bytes and is used to store the protocol header and data.

[0014] During the conversion process of the structure linked list of DPDK storing data packets into the structure linked list of the user-mode protocol stack, the control information of the structure mbuf header is updated, and a new structure is inserted into the head of the structure linked list of the user-mode protocol stack to mark the header protocol of the batch of data packets.

[0015] Furthermore, when the client and the server interact with each other, a breakpoint retransmission strategy is set to save the breakpoint information that occurs during the transmission process. The breakpoint retransmission strategy includes batch breakpoint retransmission and single file breakpoint retransmission. The single file breakpoint retransmission is as follows:

[0016] Set up window tables in the client and server to save breakpoints;

[0017] The client parses the instruction sent by the server, obtains the temporary file that needs to be retransmitted, reads the segment where the file breakpoint is located in the temporary file, and encapsulates it into a message for sending;

[0018] The server parses the message and fragments the parsed file, mounts the fragmented file to the fragment list, traverses the fragment list, discards the fragments that have been transmitted, and replies to the client with the task number and the maximum number of connections that can be established;

[0019] After receiving the maximum number of connections and the task number, the client first creates a blank temporary file according to the size, establishes a connection according to the maximum number of connections, and sends a message requesting fragmentation;

[0020] After receiving the fragmentation request message, the server extracts the fragments on the fragmentation list according to the multi-process, and first sends the current file fragment information, and then sends the file fragment content;

[0021] The client writes the file segment contents into a blank temporary file in multiple processes simultaneously according to the file segment information;

[0022] The server sends the last fragment with the CMD_DONE command;

[0023] The client receives the CMD_DONE command, completes the transmission after writing all the file segment contents, changes the temporary file to the official file name, and implements single file breakpoint retransmission.

[0024] Furthermore, window tables are set in the client and server to save breakpoints, specifically:

[0025] When sending, the server stores the task and the corresponding fragment sequence number in the sending window table. The fragment sequence number is sent together with the file fragment information.

[0026] When the client transmits, a sending and receiving window table is generated to save the fragment sequence number received by each task;

[0027] When the client reads or writes a file, it first writes the file segment content into the local memory. When refreshing the local memory, it takes the segment sequence number corresponding to the maximum value of the continuous sequence number and writes it into the breakpoint file to save the breakpoint.

[0028] Furthermore, batch breakpoint retransmission is as follows:

[0029] (1) The client receives command line instructions, parses the instructions to separate valid configuration information, configures the parameters, writes the relative path of the target server into the message, and sends it to the server. The server finds the target path location based on the received message, obtains all file names in the target folder, concatenates the file name with the target path, writes it into the message, and sends it back to the client;

[0030] (2) The client converts the file name returned by the message into the format of local directory plus file name;

[0031] (3) Check if the file name list is empty. If it is not empty, go to step (4). If it is empty, go to step (5).

[0032] (4) Search the original directory of the client to see if the file exists. If so, proceed to step (6); if not, proceed to step (7);

[0033] (5) Batch breakpoint retransmission ends;

[0034] (6) Indicates that the file has been downloaded, skip it, and go to step (3);

[0035] (7) Search whether the file exists under fuzzy query. If it exists, go to step (8); if not, go to step (9);

[0036] (8) The file needs to be retransmitted at a breakpoint. The multithreading starts a single file retransmission service based on a single file breakpoint retransmission cycle. After the breakpoint transmission is completed, it goes to step (3).

[0037] (9) The single file transfer process is started directly based on the single file breakpoint retransmission, and after the breakpoint transmission is completed, it goes to step (3).

[0038] Furthermore, when a small file is transmitted between the server and the client, the small file is a file below 32MB, and the transmission is performed using the following transmission strategy:

[0039] When the server process starts, a server subprocess is started. When the client initiates a small file transfer request to the server, a client subprocess is started at the same time.

[0040] The server receives and parses the instruction information of the small file transfer request. The server main process finds the directory where the small file is located based on the instruction information, traverses the small files in the directory, and sends the traversed small files to the server sub-process for file merging. The small files are merged into file blocks. When the first file block is generated, it initiates transmission to the client, and the server sub-process continues to merge other files.

[0041] The client receives the file block and writes it to the corresponding directory. When a file block is written, it is handed over to the client subprocess for decompression;

[0042] After the server completes the transmission, it sends a completion command to achieve the transmission of small files.

[0043] Furthermore, after writing a file block, it is handed over to the client subprocess for decompression. The client subprocess decompresses the file block as follows:

[0044] Block the client subprocess and wait for information from the server main process;

[0045] After receiving the information from the server main process, the client releases the blocking state of the client subprocess and decomposes the received file blocks;

[0046] Check whether all file blocks have been decomposed. If so, the client subprocess enters the blocking state again and waits for the next release instruction. If not, continue to decompose the file blocks.

[0047] Furthermore, a multi-mode file transfer method based on the full user-mode QUIC protocol includes a client and a server, wherein the server includes an application, a QUIC protocol, a libevent event notification library, a user-mode protocol stack and a DPDK, and the multi-mode file transfer method includes the following steps:

[0048] S01, the application of the server sends a data packet to the application layer of the server. The QUIC protocol of the application layer first initializes the event and sets the callback function of the event, adds the initialized event to the libevent event notification library of the server, completes the event registration, and libevent mounts the initialized event in the libevent event notification library to the waiting list, enters an infinite loop, and waits for the event to be triggered. The data packet parsed from the structure list of the user-mode protocol stack triggers the event in the waiting list. The triggered event is separated from the waiting list and becomes a ready event. Libevent calls the callback function of the ready event to write the data packet to the user-mode protocol stack of the server;

[0049] S02, converting the structure linked list of the user-mode protocol stack that writes the data packet into the structure linked list of the DPDK storing the data packet, the DPDK on the server side parses the data packet from the structure linked list of the DPDK storing the data packet by polling, and stores the parsed data packet into the local buffer of the network card on the server side;

[0050] S03. The network card of the server sends a data packet to the client to complete the transmission of file information from the server to the client.

[0051] The advantages of the multi-mode file transfer method based on the full user-mode QUIC protocol provided by the present invention are: the multi-mode file transfer method based on the full user-mode QUIC protocol provided in the structure of the present invention starts from the underlying data packet transceiver architecture of the operating system, uses the DPDK framework in the underlying data packet transceiver architecture, separates the data layer from the control layer, and the kernel is only responsible for processing part of the control instructions, and the protocol stack parsing, memory management, processor scheduling and other operations are all performed in the user space. In addition, this embodiment uses a user-mode protocol stack, and the data packets received by DPDK polling are directly handed over to the user-mode protocol stack for parsing, thereby avoiding frequent copying of data packets between the kernel space and the user space. The data packet parsed by the protocol stack is passed to the application layer QUIC protocol via the high-performance event library libevent, thereby realizing the QUIC protocol in full user mode; in addition, for small file transfer scenarios, consider using file aggregation as a way to alleviate file reading and writing and frequent switching of tasks in file transfer to rebuild connections, and use pipeline method to improve the transmission efficiency of the system. Use a process dedicated to file aggregation to aggregate files. Each time a file block is aggregated, a semaphore is provided to the transmission process to allow the transmission process to transmit. The transmission and aggregation are carried out simultaneously to improve system utilization and transmission efficiency and avoid stop-wait problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the process of the present invention;

[0053] Figure 2 It is a schematic diagram of the transmission framework;

[0054] Figure 3 It is a schematic diagram of the overall transmission process;

[0055] Figure 4 This is a schematic diagram of information transmission between the network card and the protocol stack;

[0056] Figure 5 It is a schematic diagram of information transmission between the protocol stack and the application layer;

[0057] Figure 6 Flowchart diagram saved for breakpoints;

[0058] Figure 7 This is a flow chart of breakpoint resume;

[0059] Figure 8 This is a flow chart of batch breakpoint retransmission;

[0060] Fig. 9 This is a flowchart of small file transfer;

[0061] Fig.10 It is a schematic diagram of the process of merging files by the server terminal process;

[0062] Fig.11 Schematic diagram of the decompression process of the file block by the client subprocess. DETAILED DESCRIPTION

[0063] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0064] In order to solve the problems existing in the background technology, this embodiment provides a file transmission method, which is implemented based on the following transmission framework: Figure 2 As shown, the transmission framework includes a client and a server, both of which have applications, QUIC protocol, libevent event notification library, user-mode protocol stack and DPDK; on the server, data is encapsulated layer by layer through the application, QUIC protocol, libevent event notification library, user-mode protocol stack and DPDK, and finally sent to the network by the server's network card; on the client, after the network card receives the data packet, it is decapsulated layer by layer through DPDK, user-mode protocol stack, libevent event notification library, and QUIC protocol, and finally the client's application obtains the data and completes the transmission.

[0065] like Figures 1 to 11 As shown, the multi-mode file transmission method based on the full user-mode QUIC protocol proposed in the present invention includes the following steps:

[0066] S1, the client requests the server for information about the file to be transferred;

[0067] S2. The client's network card obtains the data packet sent by the server and stores it in the local buffer. The client's DPDK obtains the data packet from the client's network card by polling, and encapsulates the data packet into the rte_mbuf structure of the structure list of the DPDK data packet storage, and converts the structure list of the DPDK data packet storage into the structure list of the user mode protocol stack;

[0068] S3. The QUIC protocol of the client's application layer first initializes the event and sets the callback function of the event, adds the initialized event to the client's libevent event notification library, completes the event registration, and libevent mounts the initialized event in the libevent event notification library to the waiting list, enters an infinite loop, and waits for the event to be triggered. The data packet parsed from the structure list of the user-mode protocol stack triggers the event in the waiting list. The triggered event is separated from the waiting list and becomes a ready event. Libevent calls the callback function of the ready event to read the data packet from the user-mode protocol stack and sends the read data packet to the client's application to enable the client to obtain file information from the server.

[0069] The following is a detailed description:

[0070] like Figure 3 As shown, the overall transmission process of this embodiment mainly includes five interactions.

[0071] 1) Before transferring files, the client needs to send a request command to the server, which mainly contains the specific information of the requested file download;

[0072] 2) The server finds the specific file according to the command sent by the client, and then splits the file, hangs the fragment information on the fragment queue, and generates the number of this transfer task. After completing the corresponding operation, it sends a response to the client, informing the size of the file to be transferred and the number of this transfer task;

[0073] 3) The client first creates a temporary file based on the transfer number, and then sends a command to the server requesting the file to be transferred. The command contains the number of the transfer task.

[0074] 4) After receiving the transmission number, the server finds the corresponding file queue according to the transmission number and takes out the file segment data block from the file queue for transmission; when transmitting segment data, multiple connections can be established between the client and the server for parallel transmission to improve transmission efficiency;

[0075] 5) After all shard data is transferred, the client sends a request to the server to close the connection, and the file transfer is completed.

[0076] (A) Full user-mode data transfer

[0077] The core innovation of this embodiment is to implement the full user-mode QUIC protocol, focusing on how to complete the transmission of data packets from the network card to the protocol stack and then to the QUIC protocol in user mode. Next, take the client receiving data as an example to explain the transmission process in detail. The server only needs to do the reverse operation, which will not be repeated.

[0078] (a1) Transmission between network card and protocol stack

[0079] The mutual transmission of data packets between the network card and the user-mode protocol stack mainly depends on DPDK. The specific process is as follows Figure 4 As shown, the main steps are as follows.

[0080] a1-1) After the network card receives the data packet, it first stores it in the local buffer and waits for DPDK to fetch the data packet;

[0081] a1-2) DPDK obtains data packets from the network card by polling, and encapsulates the data packets into the rte_mbuf structure. The rte_mbuf structure header stores flag information, such as address, packet length, data length, etc., and then reserves head and tail space in the payload for user customization, and the rest of the space is used to store data packets. For giant long data packets, multiple rte_mbuf structures can be linked for storage.

[0082] a1-3) Convert the structure linked list of DPDK storing data packets into the structure linked list of user-mode protocol stack; this embodiment selects the protocol stack of FreeBSD system, which uses the structure mbuf to store data packets. The structure mbuf consists of a header and a data area. The header occupies 20 bytes and contains control information such as the linked list pointer and data length. The data area consists of 108 bytes and is mainly used to store protocol headers and data. In the process of converting the structure linked list of DPDK storing data packets into the structure linked list of user-mode protocol stack, it is necessary to update the control information of the structure mbuf header and insert a new structure into the header of the structure linked list of user-mode protocol stack to mark the header protocol of this batch of data packets;

[0083] a1-4) The user-mode protocol stack parses the data packet; according to the header protocol information stored in the header node of the structure linked list, the header of the data packet is stripped off layer by layer, and finally the application layer message is obtained.

[0084] The full user-mode QUIC protocol implemented under the DPDK framework proposed in this embodiment is more efficient than existing file transfer protocols, has faster data packet processing speed, higher single connection throughput, and significantly improved overall speed of large file transfer.

[0085] (a2) Transfer between protocol stack and application layer

[0086] The mutual transmission of data packets between the protocol stack and the application layer mainly relies on libevent, which is a lightweight, open source, high-performance event notification library written in C language. The specific process is as follows Figure 5 As shown, the main steps are as follows.

[0087] a2-1) The QUIC protocol of the application layer first initializes the event and sets the corresponding callback function, which mainly includes the reading / retrieving operations of the data packet;

[0088] a2-2) Add the initial event to the libevent event notification library to complete the event registration;

[0089] a2-3) libevent mounts the initialized events in the libevent event notification library into the waiting list, enters an infinite loop, and waits for the event to be triggered;

[0090] a2-4) After the user-mode protocol stack parses the data packet, it triggers the event in the waiting list;

[0091] a2-5) The triggered event will be removed from the waiting list and become a ready event;

[0092] a2-6) libevent calls the callback function of the ready event to perform event processing. It reads data packets from the user-mode protocol stack and sends the read data packets to the client application to enable the client to obtain file information from the server;

[0093] If the server sends data, libevent calls the callback function of the ready event to call write to write the data packet to the user mode protocol stack.

[0094] This embodiment analyzes the drawbacks of the current file transfer protocol based on the actual file transfer scenario, innovatively uses the DPDK architecture and user-mode protocol stack to avoid the standard QUIC protocol's reliance on the kernel, and implements the QUIC protocol in full user mode. All data packet processing operations are migrated to user space, avoiding frequent system calls, network card interruptions, and a large number of data copies between kernel space and user space, thereby greatly improving the data packet processing speed. Especially for large file transfer tasks, it is necessary to process a large number of long data packets.

[0095] (B) Breakpoint retransmission

[0096] Since the current network environment fluctuates greatly and there are many emergencies, and the transmission scenario of scientific big data is also a long-distance, large-volume transmission environment, the transmission may fail due to the failure of one or several transmission nodes during the transmission process, and due to the huge amount of data, retransmission also has a lot of overhead waste. Therefore, this embodiment adds a breakpoint retransmission strategy. Breakpoint retransmission is divided into batch breakpoint retransmission and single file breakpoint retransmission according to the transmission target.

[0097] (b1) Single file breakpoint retransmission: Breakpoint retransmission first needs to save the breakpoint information. Considering that the breakpoint retransmission requirement in file transmission is generally due to extreme situations such as line failure or server failure, in this case, the breakpoint is best saved locally on the server. Figure 6 and 7 As shown, the single file breakpoint retransmission is as follows:

[0098] b1-1) Set up window tables in the client and server to save breakpoints;

[0099] like Figure 6 As shown, when sending, the server stores the task and the corresponding fragment sequence number into the sending window table, and the fragment sequence number is sent together with the file fragment information; when the client transmits, a sending and receiving window table is generated to save the fragment sequence number received by each task; when the client reads and writes the file, the file fragment content is first written into the local memory. When refreshing the local memory, the fragment sequence number corresponding to the maximum value of the continuous sequence number is taken and written into the breakpoint file to achieve breakpoint preservation.

[0100] After saving the breakpoint, breakpoint retransmission becomes possible. The breakpoint resume in this article is based on this to perform breakpoint retransmission.

[0101] b1-2) The client parses the instruction sent by the server, obtains the temporary file that needs to be retransmitted, reads the segment where the file breakpoint is located in the temporary file, and encapsulates it into a message for sending.

[0102] b1-3) The server parses the message and fragments the parsed file, mounts the fragmented file to the fragment list, traverses the fragment list, discards the fragments that have been transmitted, and replies to the client with the task number and the maximum number of connections that can be established;

[0103] After receiving the message, the server goes to the destination address to retrieve the file and fragments the file according to the set size to achieve file fragmentation.

[0104] b1-4) The client accepts and establishes a connection: After receiving the maximum number of connections and the task number, the client first creates a blank temporary file according to the size, establishes a connection according to the maximum number of connections, and sends a message requesting fragmentation;

[0105] b1-5) The server sends the fragment: After receiving the fragment request message, the server extracts the fragment on the fragment list according to the multi-process, and first sends the current file fragment information, and then sends the file fragment content;

[0106] b1-6) Client receives the fragments: The client writes the file fragment contents into a blank temporary file in multiple processes simultaneously according to the file fragment information;

[0107] b1-7) The server completes sending and the client completes transmission: The server sends the last segment with the CMD_DONE instruction; the client receives the CMD_DONE instruction, completes the transmission after writing all the file segment contents, and changes the temporary file to the official file name, realizing single file breakpoint retransmission.

[0108] (b2) Batch breakpoint retransmission, such as Figure 8 shown.

[0109] (1) The client receives command line instructions, parses the instructions to separate valid configuration information, configures the parameters, writes the relative path of the target server into the message, and sends it to the server. The server finds the target path location based on the received message, obtains all file names in the target folder, concatenates the file name with the target path, writes it into the message, and sends it back to the client;

[0110] (2) The client converts the file name returned by the message into the format of local directory plus file name;

[0111] (3) Check if the file name list is empty. If it is not empty, go to step (4). If it is empty, go to step (5).

[0112] (4) Search the original directory of the client to see if the file exists. If so, proceed to step (6); if not, proceed to step (7);

[0113] (5) Batch breakpoint retransmission ends;

[0114] (6) Indicates that the file has been downloaded, skip it, and go to step (3);

[0115] (7) Search whether the file exists under fuzzy query. If it exists, go to step (8); if not, go to step (9);

[0116] (8) The file needs to be retransmitted at a breakpoint. The multithreading starts a single file retransmission service based on a single file breakpoint retransmission cycle. After the breakpoint transmission is completed, it goes to step (3).

[0117] (9) The single file transfer process is started directly based on the single file breakpoint retransmission, and after the breakpoint transmission is completed, it goes to step (3).

[0118] Through the above breakpoint retransmission, the overhead of retransmitting batch data packets required in traditional breakpoint retransmission is reduced, the retransmission overhead cost is reduced, and the retransmission efficiency is improved.

[0119] (C) Small file transfer optimization strategy

[0120] The transmission of small files has also become a major obstacle to file transmission. The development of applications such as artificial intelligence, the Internet of Things, smart finance, smart security, sensor networks, and scientific experiments has generated a large amount of data, which is of various types and sizes. It includes not only a large number of large files (greater than 1GB), but also a large number of small files (less than 32MB). In particular, the number of small files will reach tens of millions, hundreds of millions, or even billions or tens of billions. When transmitting small files, we also face the following difficulties:

[0121] 1) Large number of files: The number of small files may reach thousands or millions, which means that file system operations such as file positioning, opening, closing, etc. are required frequently, which consumes a lot of time and resources.

[0122] 2) Small file size: The size of each file of a large number of small files is usually between a few KB and a few MB, which means that the transmission time of each file is very short, while the establishment and disconnection of the network connection takes a long time, which reduces the transmission efficiency.

[0123] 3) Complex file access patterns: A large number of small files often need to be read or written randomly, which means seeking and indexing in different storage locations, which increases latency and overhead.

[0124] Therefore, since most data is composed of a large number of small files, the original batch transmission method will require repeated connection establishment, resulting in low transmission rate. Fig. 9 As shown, this embodiment has an optimized transmission method for the special case of transmitting a large number of small files, and can transmit a large number of small files at high speed.

[0125] The small file transfer process is as follows:

[0126] (c1) When the server process starts, a server subprocess is started. When the client initiates a small file transfer request to the server, a client subprocess is started at the same time.

[0127] (c2) The server receives and parses the instruction information of the small file transfer request. The server main process finds the directory where the small file is located based on the instruction information, traverses the small files in the directory, and sends the traversed small files to the server sub-process for file merging. The small files are merged into file blocks. When the first file block is generated, it initiates transmission to the client, and the server sub-process continues to merge other files.

[0128] (c3) The client receives the file block and writes it to the corresponding directory. When a file block is written, it is handed over to the client subprocess for decompression;

[0129] The client subprocess decompresses the file block as follows:

[0130] c3-1) Block the client subprocess and wait for information from the server main process;

[0131] c3-2) After receiving the information from the server main process, the client releases the blocking state of the client subprocess and decomposes the received file blocks;

[0132] c3-3) Check whether all file blocks have been decomposed. If so, proceed to step c3-4). If not, proceed to step c3-5)

[0133] c3-4) The client subprocess enters the blocking state again, waiting for the next release instruction.

[0134] c3-5) continues to decompose the file blocks.

[0135] (c4) After the server completes the transmission, it sends a completion command to achieve the transmission of the small file.

[0136] The small file transmission optimization strategy proposed in this embodiment reduces the system cost caused by switching between tasks and repeatedly re-establishing connections, greatly improves the bandwidth utilization during transmission, and reduces the reading and writing costs of small files. Compared with the existing traditional file sending technology, the aggregate sending technology used in this embodiment has a faster reading and writing data processing speed, higher bandwidth utilization, faster transmission rate, fewer connections and tasks to establish, and greatly improved efficiency.

[0137] Through the above (A) to (C), the full user-mode QUIC protocol implemented in this embodiment integrates DPDK, user-mode protocol stack and libevent event notification library to realize the path of data packets from network card to QUIC protocol full user-mode, without system call and kernel state switching, achieving the effect of bypassing kernel, realizing full user-mode QUIC protocol, greatly reducing the dependence on operating system kernel, more convenient and efficient, optimizing data packet processing flow from the bottom layer, and significantly improving the efficiency of large file transmission. In addition, the small file transmission is optimized to address the pain points of small file transmission. By using the aggregation sending strategy, massive small files are merged into several large files for transmission, and then split, which greatly reduces the problem of low link utilization rate in task switching.

[0138] This embodiment starts from the underlying data packet transceiver architecture of the operating system, uses the DPDK framework in the underlying data packet transceiver architecture, separates the data layer from the control layer, and the kernel is only responsible for processing some control instructions. Protocol stack parsing, memory management, processor scheduling and other operations are all performed in the user space. In addition, this embodiment uses a user-mode protocol stack, and the data packets received by DPDK polling are directly handed over to the user-mode protocol stack for parsing, thereby avoiding frequent copying of data packets between kernel space and user space. The data packets parsed by the protocol stack are passed to the application layer QUIC protocol via the high-performance event library libevent, thereby realizing the QUIC protocol in full user mode.

[0139] This embodiment is aimed at small file transfer scenarios, and considers file aggregation as a way to alleviate file reading and writing and frequent switching of tasks in file transfer to rebuild connections, and uses a pipeline method to improve the transmission efficiency of the system. A process dedicated to file aggregation is used to aggregate files. Each time a file block is aggregated, a semaphore is provided to the transmission process to allow the transmission process to perform the transmission. The transmission and aggregation are carried out simultaneously, which improves system utilization and transmission efficiency and avoids stop-and-wait problems.

[0140] As an embodiment, to build a client-server architecture, a machine is required as a server to provide file download services, and one or more machines are required as clients. Both the client and the server (server) need to install the corresponding software and environment in advance, including applications, QUIC protocols, libevent event notification libraries, user-mode protocol stacks, and DPDK toolkits. Among them, the file transfer application in the application is recommended to be built by yourself, and it is best to introduce the file segmentation function to achieve parallel transmission and maximize the transmission efficiency; the QUIC protocol can choose the standard version of QUIC open sourced by IETF and non-standard versions of QUIC such as Facebook's open source MVFST and Alibaba's open source XQUIC. Both the libevent event notification library and DPDK have a complete open source community and can also be obtained from open source code communities such as Github. The user-mode protocol stack can be built by itself or directly use open source code. The only requirement is that it contains three and four-layer protocol stack functions and can parse IP and UDP packets.

[0141] After obtaining the basic software resources, configure the code and build a complete full user-mode QUIC protocol. If the entire process of data packet processing can be completed in user space, it means that the environment is successfully built. In addition, before specific use, the network card that supports DPDK needs to be uninstalled and bound to the user-mode driver provided by DPDK. After the network card is bound, all configuration work has been completed. The client and the server can communicate according to the process described in this embodiment and transfer the specified files.

[0142] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-mode file transfer method based on the full user-mode QUIC protocol, characterized in that: The multi-mode file transfer method includes a client and a server. The client includes an application, a QUIC protocol, a libevent event notification library, a user-mode protocol stack, and DPDK. The multi-mode file transfer method includes the following steps: S1, the client requests the server for information about the file to be transferred; S2. The client's network card obtains the data packet sent by the server and stores it in the local buffer. The client's DPDK obtains the data packet from the client's network card by polling, and encapsulates the data packet into the rte_mbuf structure of the structure list of the DPDK data packet storage, and converts the structure list of the DPDK data packet storage into the structure list of the user mode protocol stack; S3. The QUIC protocol of the client's application layer first initializes the event and sets the callback function of the event, adds the initialized event to the client's libevent event notification library, completes the event registration, and libevent mounts the initialized event in the libevent event notification library to the waiting list, enters an infinite loop, and waits for the event to be triggered. The data packet parsed from the structure list of the user-mode protocol stack triggers the event in the waiting list. The triggered event is separated from the waiting list and becomes a ready event. Libevent calls the callback function of the ready event to read the data packet from the user-mode protocol stack and sends the read data packet to the client's application, so that the client can obtain file information from the server. When the client and the server interact with each other, a breakpoint retransmission strategy is set to save the breakpoint information that occurs during the transmission process. When the breakpoint retransmission strategy is single-file breakpoint retransmission, the retransmission strategy is as follows: Set up window tables in the client and server to save breakpoints; The client parses the instruction sent by the server, obtains the temporary file that needs to be retransmitted, reads the segment where the file breakpoint is located in the temporary file, and encapsulates it into a message for sending; The server parses the message and fragments the parsed file, mounts the fragmented file to the fragment list, traverses the fragment list, discards the fragments that have been transmitted, and replies to the client with the task number and the maximum number of connections that can be established; After receiving the maximum number of connections and the task number, the client first creates a blank temporary file according to the size, establishes a connection according to the maximum number of connections, and sends a message requesting fragmentation; After receiving the fragmentation request message, the server extracts the fragments on the fragmentation list according to the multi-process, and first sends the current file fragment information, and then sends the file fragment content; The client writes the file segment contents into a blank temporary file in multiple processes simultaneously according to the file segment information; The server sends the last fragment with the CMD_DONE command; The client receives the CMD_DONE command, completes the transmission after writing all the file segment contents, changes the temporary file to the official file name, and implements single file breakpoint retransmission.

2. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 1 is characterized in that: The rte_mbuf structure header stores flag information, which includes address, packet length and data length. Head and tail space is reserved in the payload part for user customization, and other space is used to store data packets. For giant long data packets, multiple rte_mbuf structures are linked and stored.

3. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 1 is characterized in that: The user-mode protocol stack uses the mbuf structure to store data packets. The mbuf structure consists of a header and a data area. The header occupies 20 bytes and contains a linked list pointer and data length. The data area consists of 108 bytes and is used to store the protocol header and data. During the conversion process of the structure linked list of DPDK storing data packets into the structure linked list of the user-mode protocol stack, the control information of the structure mbuf header is updated, and a new structure is inserted into the head of the structure linked list of the user-mode protocol stack to mark the header protocol of the batch of data packets.

4. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 1 is characterized in that: Set up window tables in the client and server to save breakpoints, specifically: When sending, the server stores the task and the corresponding fragment sequence number in the sending window table. The fragment sequence number is sent together with the file fragment information. When the client transmits, a sending and receiving window table is generated to save the fragment sequence number received by each task; When the client reads or writes a file, it first writes the file segment content into the local memory. When refreshing the local memory, it takes the segment sequence number corresponding to the maximum value of the continuous sequence number and writes it into the breakpoint file to save the breakpoint.

5. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 1 is characterized in that: When the breakpoint retransmission strategy is batch breakpoint retransmission, the specific retransmission strategy is as follows: (1) The client receives command line instructions, parses the instructions to separate valid configuration information, configures the parameters, writes the relative path of the target server into the message, and sends it to the server. The server finds the target path location based on the received message, obtains all file names in the target folder, concatenates the file name with the target path, writes it into the message, and sends it back to the client; (2) The client converts the file name returned by the message into the format of local directory plus file name; (3) Check if the file name list is empty. If it is not empty, go to step (4). If it is empty, go to step (5). (4) Search the original directory of the client to see if the file exists. If so, proceed to step (6); if not, proceed to step (7); (5) Batch breakpoint retransmission ends; (6) Indicates that the file has been downloaded, skip it, and go to step (3); (7) Search whether the file exists under fuzzy query. If it exists, go to step (8); if not, go to step (9); (8) The file needs to be retransmitted at a breakpoint. The multithreading starts a single file retransmission service based on a single file breakpoint retransmission cycle. After the breakpoint transmission is completed, it goes to step (3). (9) The single file transfer process is started directly based on the single file breakpoint retransmission, and after the breakpoint transmission is completed, it goes to step (3).

6. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 1, characterized in that: When a small file is transferred between the server and the client, the small file is a file below 32MB and is transferred using the following transfer strategy: When the server process starts, a server subprocess is started. When the client initiates a small file transfer request to the server, a client subprocess is started at the same time. The server receives and parses the instruction information of the small file transfer request. The server main process finds the directory where the small file is located based on the instruction information, traverses the small files in the directory, and sends the traversed small files to the server sub-process for file merging. The small files are merged into file blocks. When the first file block is generated, it initiates transmission to the client, and the server sub-process continues to merge other files. The client receives the file block and writes it to the corresponding directory. When a file block is written, it is handed over to the client subprocess for decompression; After the server completes the transmission, it sends a completion command to achieve the transmission of small files.

7. The multi-mode file transfer method based on the full user-mode QUIC protocol according to claim 6 is characterized in that: After writing a file block, it is handed over to the client subprocess for decompression. The client subprocess decompresses the file block as follows: Block the client subprocess and wait for information from the server main process; After receiving the information from the server main process, the client releases the blocking state of the client subprocess and decomposes the received file blocks; Check whether all file blocks have been decomposed. If so, the client subprocess enters the blocking state again and waits for the next release instruction. If not, continue to decompose the file blocks.

8. A multi-mode file transfer method based on the full user-mode QUIC protocol, characterized in that: The multi-mode file transfer method includes the following steps: S01, the application of the server sends a data packet to the application layer of the server. The QUIC protocol of the application layer first initializes the event and sets the callback function of the event, adds the initialized event to the libevent event notification library of the server, completes the event registration, and libevent mounts the initialized event in the libevent event notification library to the waiting list, enters an infinite loop, and waits for the event to be triggered. The data packet parsed from the structure list of the user-mode protocol stack triggers the event in the waiting list. The triggered event is separated from the waiting list and becomes a ready event. Libevent calls the callback function of the ready event to write the data packet to the user-mode protocol stack of the server; S02, converting the structure linked list of the user-mode protocol stack that writes the data packet into the structure linked list of the DPDK storing the data packet, the DPDK on the server side parses the data packet from the structure linked list of the DPDK storing the data packet by polling, and stores the parsed data packet into the local buffer of the network card on the server side; S03, the network card of the server sends a data packet to the client to complete the transmission of file information from the server to the client; When the client and the server interact with each other, a breakpoint retransmission strategy is set to save the breakpoint information that occurs during the transmission process. When the breakpoint retransmission strategy is single-file breakpoint retransmission, the retransmission strategy is as follows: Set up window tables in the client and server to save breakpoints; The client parses the instruction sent by the server, obtains the temporary file that needs to be retransmitted, reads the segment where the file breakpoint is located in the temporary file, and encapsulates it into a message for sending; The server parses the message and fragments the parsed file, mounts the fragmented file to the fragment list, traverses the fragment list, discards the fragments that have been transmitted, and replies to the client with the task number and the maximum number of connections that can be established; After receiving the maximum number of connections and the task number, the client first creates a blank temporary file according to the size, establishes a connection according to the maximum number of connections, and sends a message requesting fragmentation; After receiving the fragmentation request message, the server extracts the fragments on the fragmentation list according to the multi-process, and first sends the current file fragment information, and then sends the file fragment content; The client writes the file segment contents into a blank temporary file in multiple processes simultaneously according to the file segment information; The server sends the last fragment with the CMD_DONE command; The client receives the CMD_DONE command, completes the transmission after writing all the file segment contents, changes the temporary file to the official file name, and implements single file breakpoint retransmission.