Method for quickly reading and forwarding network range data based on zero copy

Abstract

The invention relates to a network range data fast reading and forwarding method based on zero copy, and belongs to the technical field of network range data extraction. According to the invention, the zero copy technology is adopted to realize rapid extraction of the network target range data, so that the data copy and CPU switching times in the transmission or sending process of the network target range data can be reduced, the data transmission efficiency is improved, and the performance of the network target range system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network range data extraction, and particularly relates to a method for quickly reading and forwarding network range data based on zero-copy. Background Art

[0002] In a network range, there are usually scenarios where a user reads configuration data or storage data of a network device or a host device and sends it to a specified external device or network. The usual method is through data interaction between user space, kernel space, and hardware. Data usually needs to be read from a hard disk or other storage devices into the kernel buffer, then copied to the user buffer, and finally sent to the network or other external devices as needed. This method will experience 2 times of DMA (Direct Memory Access) copy, 2 times of CPU copy, and 4 times of switching between CPU user mode and kernel mode. Summary of the Invention

[0003] (1) Technical Problems to be Solved

[0004] The technical problem to be solved by the present invention is: how to achieve fast extraction of network range data and improve the performance of the network range system.

[0005] (2) Technical Solutions

[0006] To solve the above technical problem, the present invention provides a method for quickly reading and forwarding network range data based on zero-copy. This method is used to read the storage data of the underlying device in the network range and forward it. Among them, Sendfile() is used to directly send the file content to the Socket buffer of the device in the kernel space, thereby achieving zero-copy.

[0007] The present invention also provides a system for implementing the above method.

[0008] (3) Advantageous Effects

[0009] In order to reduce the number of data copies and CPU switches during the transmission or sending of network range data and improve the data transmission efficiency, the present invention uses zero-copy technology to achieve fast extraction of network range data and improve the performance of the network range system. Among them, the present invention uses the Sendfile() method to directly send the file content to the Socket buffer of the device in the kernel space, which can effectively reduce the number of switches between user mode and kernel mode and data copies. When calling Sendfile(), only two switches are required (switching from user mode to kernel mode to call Sendfile(), and switching from kernel mode to user mode to complete the Sendfile() call), two DMA copies, and one CPU copy. Description of the Drawings

[0010] Figure 1 It is a block diagram of the implementation principle of the method of the present invention;

[0011] Figure 2 It is a flowchart of the method of the present invention;

[0012] Figure 3 It is a schematic diagram of fast data reading and forwarding realized by a zero-copy method in a network range provided by an embodiment of the present invention. Specific embodiments

[0013] To make the objectives, content, and advantages of the present invention clearer, the following further describes in detail the specific embodiments of the present invention with reference to the accompanying drawings and embodiments.

[0014] To reduce the number of data copies and CPU switches during the transmission or sending of network range data and improve data transmission efficiency, the present invention uses the zero-copy technology to realize fast extraction of network range data and improve the performance of the network range system.

[0015] The present invention provides a method for fast reading and forwarding of network range data based on zero-copy, which is used to realize fast reading of storage data of underlying devices in a network range and perform fast forwarding. As Figure 1 shown, the present invention uses the Sendfile() method to directly send the file content to the Socket buffer of the device in the kernel space, avoiding additional copies between the user space and the kernel space, thereby realizing zero-copy.

[0016] The Sendfile() system call requires two parameters, an input file descriptor in_fd and an output file descriptor out_fd; the input file descriptor in_fd is usually the file to be sent, and the output file descriptor out_fd is usually a network socket (Socket); when calling Sendfile(), the kernel directly reads data from the file buffer corresponding to the input file descriptor and sends it to the Socket buffer corresponding to the output file descriptor by means of CPU copy, and the whole process is completed in the kernel space without copying the data to the user space.

[0017] The present invention can effectively reduce the number of switches and data copies between the user space and the kernel space by using the Sendfile() method. Calling Sendfile() only requires two switches (switching from the user space to the kernel space to call Sendfile(), and switching from the kernel space to the user space to complete the Sendfile() call), two DMA copies, and one CPU copy.

[0018] As Figure 2As shown, the method of the present invention is implemented on the user host, and it is necessary to copy the data on the storage server a of the storage cluster of the network range partition A to the storage server b of the storage cluster of the network range partition B. The business process of using the Sendfile() method to implement zero copy is:

[0019] 1. Reading process

[0020] (1) User sends a sendfile() call

[0021] When the user thread on the user host needs to read a file, it initiates a read request to the operating system of storage server a by calling Sendfile(in_fd, out_fd), and then switches from user state to kernel state;

[0022] (2) The DMA controller on the user host initiates an IO request

[0023] The DMA controller initiates an IO request to establish a transmission link for the input file descriptor in_fd;

[0024] (3) DMA copies data to the page cache

[0025] After the DMA copy request is passed, the operating system of storage server a will use the DMA controller to copy the data from the disk (or other storage device) of storage server a to the page cache (a type of file cache) corresponding to the file descriptor in_fd. This process is completed by the DMA controller and does not require CPU participation, thus reducing the CPU burden.

[0026] (4) Copying is completed and the transfer completion (finish) signal is sent

[0027] After the DMA controller completes the data copy, the DMA controller sends a transfer completion signal to the disk (or other storage device) of storage server a;

[0028] (5) Kernel CPU copies data

[0029] After the data is copied to the page cache in the kernel buffer by the DMA controller, the CPU of the user host directly copies the data from the kernel buffer (file buffer area-page cache) to the network socket buffer corresponding to the output file descriptor out_fd; at this point, the data copy process has been completed;

[0030] 2. Writing process

[0031] (1) The DMA controller initiates an IO request again

[0032] The DMA controller initiates an IO request again to establish a request link for transmitting the output file;

[0033] (2) DMA controller copies data

[0034] The DMA controller copies the data to the port of storage server b;

[0035] (3) The DMA controller completes the copy and sends a finish signal

[0036] After the DMA controller completes the data copy, it sends a transfer completion signal to the port of storage server b;

[0037] (4)Sendfile() call completed

[0038] When the data is sent, the user host returns to the user thread, switching from kernel mode to user mode to complete the data forwarding.

[0039] Example

[0040] like Figure 3 As shown, the data on the network target range partition 2 storage cluster - storage server 4 is copied through zero copy technology and forwarded to the network target range partition 1 storage cluster - storage server 4. The process is divided into two steps: reading and writing. The reading process requires 5 steps and the writing process requires 4 steps.

[0041] (1) Reading process

[0042] First, when the user thread on the user host needs to read a file, it calls the sendfile() function to initiate a read request to the storage server 4 operating system, and the user state is switched to the kernel state.

[0043] Secondly, the DMA controller initiates an IO request to establish a transmission link for the input file descriptor of the Sendfile() function;

[0044] Again, after the DMA copy request, the operating system will use the DMA controller to copy the data from the storage server 4 disk to the file buffer corresponding to the file descriptor in_fd. This process is completed by the DMA controller and does not require CPU participation, reducing the CPU burden;

[0045] Again, after the DMA controller completes the data copy, the DMA controller sends a transfer completion signal to the hardware;

[0046] Finally, after the data is copied to the page cache in the kernel buffer by DMA, the CPU directly copies the data from the kernel buffer to the socket buffer of the network socket corresponding to the output file descriptor out_fd; at this time, the data copy process is completed (which is the input process, that is, I (Input) in IO; next, the data forwarding process is executed (which is the output process, that is, O (Output) in IO;

[0047] (2) Writing process

[0048] First, the DMA controller initiates an IO request again to establish a request link for transmitting the output file;

[0049] Again, the DMA controller copies the data to Storage Server 4 of the storage cluster in Network Range Partition 1;

[0050] Again, the DMA controller sends a transmission completion finish signal to Storage Server 4 of the storage cluster in Network Range Partition 1;

[0051] Finally, after the data is sent, the user host will return to the user thread, and at this time, the kernel mode is switched to the user mode, thus completing the data forwarding work.

[0052] The above entire process experiences 2 DMA copies, 1 CPU copy, and 2 switches between the user mode and the kernel mode, greatly reducing the data copy time, reducing the data traffic in a large-scale network range environment, and improving the data reading and forwarding efficiency.

[0053] Table 1 below shows the effect differences between the traditional copy method and the method of implementing zero copy using the Sendfile system call in the present invention. By comparison, it can be seen that by using the method of the present invention, the CPU copy efficiency is increased by 50%, and the user mode switch efficiency is increased by 50%, greatly improving the system's data reading and forwarding ability.

[0054] Table 1

[0055]

[0056]

[0057] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A method for fast reading and forwarding of network range data based on zero-copy, characterized in that This method is used to read the storage data of the underlying device in the network range and forward it. Among them, Sendfile() is used to directly send the file content to the socket buffer of the device in the kernel space, so as to achieve zero-copy.

2. The method according to claim 1, wherein The Sendfile() system call requires two parameters, the input file descriptor in_fd and the output file descriptor out_fd. The input file descriptor in_fd corresponds to the file to be sent, and the output file descriptor out_fd corresponds to a network socket buffer. When calling Sendfile(), the kernel directly reads the data from the file buffer corresponding to the input file descriptor and sends it to the socket buffer corresponding to the output file descriptor by means of CPU copying. The whole process is completed in the kernel space.

3. The method according to claim 2, wherein This method is implemented on the user host and is used to copy the data on the storage server a of the storage cluster in network range partition A to the storage server b of the storage cluster in network range partition B. The specific process of using Sendfile() to implement data reading and forwarding is as follows: I. Reading process (1) The user sends a sendfile() call When the user thread on the user host needs to read a file, it sends a read request to the operating system of the storage server a by calling Sendfile(in_fd, out_fd). At this time, the user mode is switched to the kernel mode. (2) The DMA controller on the user host initiates an IO request The DMA controller initiates an IO request to establish a transmission link for the input file descriptor in_fd. (3) The DMA copies the data to the page cache After the DMA copy request, the operating system of the storage server a uses the DMA controller to copy the data from the storage device of the storage server a to the page cache corresponding to the file descriptor in_fd. (4) After the copy is completed, a transmission completion signal is sent After the DMA controller completes the data copy, the DMA controller sends a transmission completion signal to the storage device of the storage server a. (5) The kernel CPU copies the data After the data is copied to the page cache in the kernel buffer by the DMA controller, the CPU of the user host directly copies the data from the page cache to the network socket buffer corresponding to the output file descriptor out_fd. At this time, the data copy process, that is, the input process, is completed. II. Writing process (1) The DMA controller initiates an IO request again The DMA controller initiates an IO request again to establish a request link for transmitting the output file. (2) The DMA controller copies the data The DMA controller copies the data to the storage server b. (3) After the DMA controller completes the copy, it sends a transmission completion signal After the DMA controller completes the data copy, it sends a transmission completion signal to the storage server b. (4) The Sendfile() call is completed When the data is sent, the user host returns to the user thread. At this time, the kernel mode is switched to the user mode, thus completing the data forwarding work, that is, the output process.

4. The method according to claim 1, characterized in that, The user host can achieve network reachability with other devices in the network range.

5. The method according to claim 4, characterized in that, The user host can achieve network reachability with other devices in the network range through a switch and a router.

6. The method according to claim 1, characterized in that, The user thread is located in the user buffer.

7. The method according to claim 6, characterized in that, There are multiple user threads.

8. The method according to any one of claims 1 to 7, characterized in that This method is applied in network security.

9. A system for implementing the method according to any one of claims 1 to 7.

10. The system according to claim 9, wherein, This system is applied in network security.

Images