A data transmission method and system based on IDM

Abstract

The present invention provides an IDM-based data transmission method and system. The method is applied to an IDM-driven data transmission system, wherein the data transmission system includes a user layer, a kernel, a network interface card (NIC), and an SSD. The user layer represents user-layer data; the kernel represents the Linux kernel; the NIC represents a network interface card (NIC), which includes DDR; and the SSD represents a solid-state memory (SSD), which includes DDR and flash. The method includes: transmitting metadata through the IDM; directly transmitting NIC data through the IDM; and transmitting user-mode libaio data through the IDM. The present invention utilizes IDM-driven technology to reduce the number of copies during data transmission, thereby improving data transmission efficiency.

Description

Technical Field

[0001] The present invention relates to the fields of data storage technology and data transmission technology, and in particular to an IDM-based data transmission method and system. Background Art

[0002] IDM (Intelligence Data Manager) is a data transmission middleware that aims to minimize the number of copies during data transmission and provide a multi-input and multi-output data platform for decoupling data receiving modules such as smart network card drivers and smart storage card drivers from data storage modules.

[0003] Traditional bio directly carries data, which needs to be transferred from user state to kernel state, while the bio constructed by IDM only carries metadata (ba information), such as Figure 2 shown.

[0004] In terms of disk write mode, traditional bio data is written directly from host DDR to flash, while IDM constructed bio data is written from storage card DDR to flash. The data on storage card DDR is written by network card through P2P mode. Figure 6 As shown, the details are as follows:

[0005] 1) Data is transferred from the network card to the kernel state to obtain the file lba address;

[0006] 2) Save the lba address to the host ddr;

[0007] 3) Map the user space host ddr to the kernel state;

[0008] 4) Transfer kernel-mode data from host DDR to flash.

[0009] Since ordinary write operations do not consider the issue of data sharing between user mode and kernel mode, ordinary write operations will require one more copy. Summary of the Invention

[0010] The purpose of the present invention is to solve the problem of multiple copies of network card data during data transmission from the network card to the memory, thereby reducing the number of data copies.

[0011] The purpose of the present invention is to provide a data transmission method and system based on IDM to solve the problems raised in the above technical background.

[0012] To achieve the above object, the present invention adopts the following technical solutions:

[0013] The object of the present invention is to provide an IDM-based data transmission method, wherein the IDM-driven data transmission system includes User, Kernel, NIC, and SSD; User is user-layer data; Kernel is the Linux kernel; the NIC is a network card, and the network card includes DDR; the SSD is a solid-state memory, and the solid-state memory includes DDR and flash; the method includes: IDM transmitting metadata, the network card supports P2P mode, IDM only transmits metadata, and the data is on the network card DDR; IDM directly transmits network card data, and the data is on the network card DDR; IDM transmits user-state libaio data, and the data is on the host DDR, and maps the user space host DDR to the kernel state.

[0014] Preferably, the IDM transmits metadata in the following specific steps:

[0015] A1: Get the file's lba address;

[0016] A2: Save the lba address to IDM's internal lba list;

[0017] A3: NIC transfers data from NIC DDR to SSD DDR through P2P;

[0018] A4: NIC transmits the address sddr_addr of the SSD DDR where the data is located to IDM;

[0019] A5: IDM constructs bio according to lba list and sddr_addr list, and marks bio.flag as IDM_BIO_P2P;

[0020] A6: Bio is transmitted to the SSD through the system API;

[0021] A7: The SSD writes data to the SSD Flash based on information such as flag, sddr_addr, lba, and data_len.

[0022] Preferably, the IDM transmits metadata, and the data is stored on the network card DDR.

[0023] Preferably, the IDM transmits metadata, which requires P2P support.

[0024] Preferably, the IDM directly transmits network card data, and the specific steps are as follows:

[0025] B1: Get the file lba address;

[0026] B2: Save the lba address to IDM internal lba list;

[0027] B3: NIC transfers data to Host DDR;

[0028] B4: NIC transfers the host DDR address hddr_addr where the data is located to IDM;

[0029] B5: IDM constructs bio according to lba list and hddr_addr list, and marks bio.flag as IDM_BIO_HOST;

[0030] B6: bio is transferred to SSD through the system API;

[0031] B7: SSD writes data to SSD Flash according to flag, hddr_addr, lba, data_len and other information.

[0032] Preferably, the IDM directly transmits network card data, and the data is on the kernel Host DDR.

[0033] Preferably, the IDM transmits user-mode libaio data, and the specific steps are as follows:

[0034] C1: Map user-mode memory to kernel-mode;

[0035] C2: The network card transfers data to the user-mapped memory;

[0036] C3: The application initiates libaio writes to user-mode data.

[0037] C4: libaio directly obtains the page address and writes the data to the SSD flash.

[0038] Preferably, the IDM transmits user-state libaio data, does not participate in disk write, provides mapping from user space to kernel space, connects to the network card, writes the data received by the network card into the mapping space, and notifies the user to initiate a libaio write operation.

[0039] Preferably, in the IDM-based data transmission method, the IDM data input device includes a network card and user-state data, and the data output device includes a DDR and an intelligent memory.

[0040] The present invention also provides an IDM data transmission system, which applies any of the above-mentioned IDM data transmission methods.

[0041] Among them, the data transmission system includes User, Kernel, NIC, and SSD; the User is user-layer data; the Kernel is the Linux kernel; the NIC is a network card, which contains DDR; the SSD is a solid-state memory, which contains DDR and flash.

[0042] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0043] 1) The purpose of the present invention is to solve the problem of multiple copies of network card data to storage during data transmission, thereby reducing the number of data copies.

[0044] 2) Provide a multi-input and multi-output data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0046] Figure 1 It is a system principle block diagram of a preferred embodiment of the present invention;

[0047] Figure 2 It is the difference between traditional bio and idm bio;

[0048] Figure 3 It is a preferred embodiment of the present invention to transmit metadata using IDM;

[0049] Figure 4 It is a preferred embodiment of the present invention that bio directly transmits network card data;

[0050] Figure 5 It is the libaio transmission mode of the preferred embodiment of the present invention;

[0051] Figure 6 This is the traditional network card receiving and storing process of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention provides an IDM-based data transmission method and system. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0053] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0054] Example 1:

[0055] The present invention provides an application example of a data transmission method based on IDM - using IDM to transmit metadata, the transmission process is as follows Figure 1 (1) shown.

[0056] 1) The network card transfers data from the network card DDR to the storage card DDR via P2P;

[0057] 2) Get the file lba address and save the lba address to IDM's internal lba list. IDM constructs bio based on the lba list and sddr_addr list.

[0058] like Figure 3 As shown in the figure, the data transmission system includes User APP, Kernel, NIC, and SSD. User APP is user-layer data; Kernel is the Linux kernel; NIC is the network card, which contains DDR; SSD is the solid-state storage, which contains DDR and flash. The data transmission steps are as follows:

[0059] 1) User layer data first obtains the lba address;

[0060] 2) Save the lba address to IDM internal lba list;

[0061] 3) The network card transfers data from the network card DDR to the storage card DDR via P2P;

[0062] 4) The network card passes the DDR address sddr_addr where the data is located to IDM;

[0063] 5) IDM constructs bio according to lba list and sddr_addr list;

[0064] 6) bio is transferred to the solid-state storage through the system API;

[0065] 7) The solid-state memory writes data to the Flash according to information such as sddr_addr, lba, and data_len.

[0066] Example 2

[0067] Figure 4 This is an application example of a data transmission method based on IDM - bio directly transmits network card data. The transmission process is as follows Figure 1 (2) shown.

[0068] 1) The network card transfers data to the Host DDR;

[0069] 2) The network card passes the host DDR address hddr_addr where the data is located to IDM;

[0070] 3) Get the file lba address and pass it to the SSD through the system API.

[0071] The data transmission system includes User APP, Kernel, NIC, and SSD. User APP is user-layer data; Kernel is the Linux kernel; NIC is the network card, which contains DDR; SSD is the solid-state storage, which contains DDR and flash. The transmission steps are as follows:

[0072] 1) User layer data first obtains the lba address;

[0073] 2) Save the lba address to IDM internal lba list;

[0074] 3) The network card transfers the data to the Host DDR;

[0075] 4) The network card passes the host DDR address hddr_addr where the data is located to IDM;

[0076] 5) IDM constructs bio according to lba list and hddr_addr list;

[0077] 6) bio is transferred to the solid-state storage through the system API;

[0078] 7) The solid-state memory writes data to the Flash according to information such as hddr_addr, lba, and data_len.

[0079] Example 3

[0080] Figure 5 This is an application example of a data transmission method based on IDM - libaio transmission method.

[0081] The data transmission system includes User APP, Kernel, NIC, and SSD. User APP is user-layer data; Kernel is the Linux kernel; NIC is the network card, which contains DDR; SSD is the solid-state memory, which contains DDR and flash. The specific transmission steps are as follows:

[0082] 1) Map user-mode memory to kernel-mode.

[0083] 2) The network card transfers the data to the user-mapped memory;

[0084] 3) The application initiates libaio writes to user-mode data;

[0085] 4) Since the host DDR is shared between user mode and kernel mode, after the network card writes data to the host DDR, the user can directly initiate a write operation, but must use libaio to write; the application initiates a libaio write operation on the user mode data, and libaio directly obtains the page address and writes the data to the flash.

[0086] Figure 6 The traditional network card receives the stored procedure, as follows:

[0087] 1) Data is transferred from the network card to the kernel state to obtain the file lba address;

[0088] 2) Save the lba address to the host ddr;

[0089] 3) Map the user space host ddr to the kernel state;

[0090] 4) Transfer kernel-mode data from host DDR to flash.

[0091] Because ordinary write operations do not consider the issue of data sharing between user mode and kernel mode, ordinary write operations will require one more copy.

[0092] In summary, the IDM-based data transmission method and system solves the problem of multiple copies of network card data during data transmission to the storage, thereby reducing the number of data copies.

[0093] While the specific embodiments of the present invention have been described in detail above, these are merely exemplary and the present invention is not limited thereto. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, any equivalent changes and modifications made without departing from the spirit and scope of the present invention are intended to be encompassed within the scope of the present invention.

Claims

1. A data transmission method based on IDM, characterized in that: The data transmission system is applied to IDM drive, and the data transmission system includes User, Kernel, NIC, and SSD; the User is user layer data; the Kernel is the Linux kernel; The NIC is a network card, which includes DDR; the SSD is a solid-state memory, which includes DDR and flash; the method includes: IDM transmitting metadata; IDM directly transmitting network card data; IDM transmitting user-state libaio data; The IDM transfers metadata in the following steps: A1: Get the file's lba address; A2: Save the lba address to IDM's internal lbalist; A3: NIC transfers data from NIC DDR to SSD DDR through P2P; A4: NIC transmits the address sddr_addr of the SSD DDR where the data is located to IDM; A5: IDM constructs bio based on lba list and sddr_addr list and marks bio.flag as IDM_BIO_P2P. A6: Bio is transmitted to the SSD through the system API; A7: The SSD writes data to the SSD Flash based on flag, sddr_addr, lba, and data_len information. The steps for IDM to directly transmit network card data are as follows: B1: Get the file lba address; B2: Save the lba address to IDM's internal lbalist; B3: NIC transfers data to Host DDR; B4: NIC transfers the host DDR address hddr_addr where the data is located to IDM; B5: IDM constructs bio according to lba list and hddr_addr list; and marks bio.flag as IDM_BIO_HOST; B6: bio is transferred to SSD through the system API; B7: SSD writes data to SSD Flash according to flag, hddr_addr, lba, and data_len information; The steps for IDM to transfer user-mode libaio data are as follows: C1: Map user-mode memory to kernel-mode; C2: The network card transfers data to the user-mapped memory; C3: The application initiates libaio writes to user-mode data. C4: libaio directly obtains the page address and writes the data to the SSD flash.

2. The data transmission method based on IDM according to claim 1, characterized in that: The IDM transmits metadata, and the metadata is stored on the network card DDR.

3. The data transmission method based on IDM according to claim 1, characterized in that: The IDM transmits metadata and requires P2P support.

4. The data transmission method based on IDM according to claim 1, characterized in that: The IDM directly transmits the network card data, and the data is on the kernel Host DDR.

5. The data transmission method based on IDM according to claim 1, characterized in that: The IDM transmits user-state libaio data and does not participate in disk write. It provides a mapping from user space to kernel space, connects to the network card, writes the data received by the network card into the mapping space, and notifies the user to initiate a libaio write operation.

6. The data transmission method based on IDM according to claim 1, characterized in that: IDM data input devices include network cards and user-mode data, and data output devices include DDR and intelligent memory.

7. The data transmission system based on IDM driver according to claim 1, characterized in that: The data transmission system includes User, Kernel, NIC, and SSD; the User is user-layer data; the Kernel is the Linux kernel; the NIC is a network card containing DDR; the SSD is a solid-state memory containing DDR and flash.

Images