A method and device for reading and writing remote memory based on RDMA transmission protocol

Abstract

The application relates to a remote memory reading and writing method and device based on an RDMA transmission protocol; the method comprises the following steps: creating a virtual character device based on memory information transmitted by a remote node, and mapping the memory information to a local virtual device; when a page fault interrupt is triggered, allocating physical memory, initiating an RDMA reading request, reading target memory data of the remote node to the physical memory, and performing write protection on the target memory data; when a process in a user state modifies the target memory data, marking the target memory data with a dirty mark; periodically detecting the local physical memory according to a preset time interval; when the target memory data carrying the dirty mark is detected, initiating an RDMA writing request, and writing back the target memory data carrying the dirty mark to the remote node; and when the physical memory is detected to be overdue, recycling the physical memory; the method can accelerate the data synchronization performance between the local node and the remote node, and improve the memory utilization efficiency.

Description

Technical Field

[0001] This application relates to the field of high-performance network technology, and in particular to a remote memory read / write method and apparatus based on the RDMA transmission protocol. Background Technology

[0002] In high-performance computing, distributed systems, database systems, and other application scenarios, data sharing and synchronization between different nodes are required. For example, when using containers to provide corresponding services, multiple image data need to be pulled from a remote location during the container initialization and startup process. When the image data is large, it will take a long time to retrieve the image data, and other processes or users will be idle during this period. In addition, not all the data retrieved is necessary data. That is, there are problems such as poor data synchronization performance between nodes and low memory utilization efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a remote memory read / write method and apparatus based on the RDMA transmission protocol that can accelerate data synchronization performance between nodes and improve memory utilization efficiency in response to the above-mentioned technical problems.

[0004] In a first aspect, this application provides a remote memory read / write method based on the RDMA transmission protocol. The method is applied to a local node, which is used to connect to a remote node. The method includes:

[0005] Based on the memory information transmitted from the remote node, a virtual character device is created, and the memory information is mapped to the virtual character device;

[0006] When a page fault is triggered, physical memory is allocated and an RDMA read request is initiated to the remote node; the RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in physical memory.

[0007] Write protection is set for the target memory data. When the target memory data is modified by a user-space process, the target memory data is marked as dirty. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry a dirty mark, an RDMA write request is initiated to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty mark to the remote node.

[0008] When a physical memory timeout is detected, the physical memory is reclaimed.

[0009] In one embodiment, a virtual character device is created based on memory information transmitted from a remote node, including:

[0010] Initiate a TCP connection establishment request to the remote node. The TCP connection establishment request is used to instruct the remote node and the local node to establish a TCP connection.

[0011] A virtual character device is created based on memory information.

[0012] In one embodiment, when a page fault is triggered, physical memory is allocated, and an RDMA read request is initiated to the remote node, including:

[0013] Supports user-space calls to the open and mmap interfaces;

[0014] Initiate an RDMA QP negotiation message to the remote node. The RDMA QP negotiation message is used to instruct the remote node and the local node to establish a QP connection.

[0015] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, if it is determined that the virtual character device has not cached the target memory data, trigger a page fault and initiate an RDMA read request.

[0016] Allocate physical memory and cache the target memory data of the remote node in physical memory.

[0017] In one embodiment, after initiating an RDMA write request to the remote node when the target memory data is detected to carry a dirty flag, the following steps are included:

[0018] Clear the dirty flag and reset write protection for the target memory data.

[0019] In one embodiment, when a physical memory timeout is detected, physical memory is reclaimed, including:

[0020] Supports user-space calls to the close and munmap interfaces;

[0021] Set the physical memory to an idle state and record the idle time;

[0022] Periodically check the status of physical memory, and reclaim physical memory when it is idle for a preset time.

[0023] Destroy the QP connection between the remote node and the local node.

[0024] In one embodiment, the virtual character device supports the open interface and the mmap interface; the method further includes:

[0025] The virtual character device is opened by calling the open interface;

[0026] The target memory data is mapped to user space by calling the mmap interface.

[0027] In one embodiment, the virtual character device supports the close interface and the munmap interface, and the method includes:

[0028] By calling the munmap interface, the mapping relationship between memory information and virtual character devices is removed;

[0029] The virtual character device is closed by calling the close interface.

[0030] Secondly, this application also provides a remote memory read / write device based on the RDMA transmission protocol. The device is applied to a local node, which is used to connect to a remote node. The device includes:

[0031] The device creation module is used to create virtual character devices based on memory information transmitted from remote nodes and to map memory information to virtual character devices.

[0032] The read module is used to allocate physical memory and initiate an RDMA read request to the remote node when a page fault is triggered. The RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in physical memory.

[0033] The write module is used to set write protection for target memory data. When the target memory data is modified by a user-space process, the target memory data is marked as dirty. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry a dirty mark, an RDMA write request is initiated to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty mark to the remote node.

[0034] The memory management module is used to reclaim physical memory when a physical memory timeout is detected.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0037] The aforementioned remote memory read / write method and apparatus based on the RDMA transfer protocol establishes a connection between a local node and a remote node. Based on the memory information transmitted by the remote node, a virtual character device is created, mapping the memory information from the local node to the virtual device. When a page fault is triggered, physical memory is allocated, an RDMA read request is initiated, and the target memory data from the remote node is read into the physical memory. Write protection is applied to the target memory data. When a user-mode process modifies the target memory data, a dirty flag is added to the target memory data. The local physical memory is periodically checked at preset time intervals. When target memory data carrying a dirty flag is detected, an RDMA write request is initiated, and the dirty flag is written to the target memory data. The application supports writing target memory data back to remote nodes and reclaiming physical memory when a physical memory timeout is detected. It allows users to customize aggregation mapping strategies to aggregate and map memory information from remote nodes to virtual devices within the local node. When a page fault is triggered, an RDMA read request is initiated to promptly read the target memory data from the remote node to the local node. When modification of the target memory data is detected, an RDMA write request is initiated to write the modified target memory data back to the remote node. This accelerates data synchronization between the local and remote nodes. Compared to traditional technologies where not all acquired data is necessary, this application supports on-demand caching of target memory data, improving memory utilization efficiency. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a block diagram of a remote memory aggregation read / write system in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a remote memory read / write method based on the RDMA transmission protocol in one embodiment.

[0041] Figure 3 This is an application scenario diagram of a remote memory read / write method based on the RDMA transmission protocol in one embodiment;

[0042] Figure 4 A flowchart illustrating the steps of creating a virtual character device for a local node in one embodiment;

[0043] Figure 5This is a flowchart illustrating the steps of allocating physical memory and initiating an RDMA read request to a remote node when a page fault is triggered in one embodiment.

[0044] Figure 6 This is a flowchart illustrating the steps of initiating an RDMA write request to a remote node when the target memory data is detected to carry a dirty flag, as shown in one embodiment.

[0045] Figure 7 This is a flowchart illustrating the steps of reclaiming physical memory after a physical memory timeout is detected in one embodiment.

[0046] Figure 8 This is a structural block diagram of a remote memory read / write device based on the RDMA transmission protocol in one embodiment.

[0047] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0050] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0051] It is understandable that "at least one" refers to one or more, and "multiple" refers to two or more. "At least a part of an element" refers to part or all of an element.

[0052] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0053] This application provides a remote memory read / write method based on the RDMA (Remote Direct Memory Access) transport protocol, which can be applied to, for example... Figure 1 The remote memory aggregation read / write system shown is configured in the local node and includes the following four modules:

[0054] The main module is responsible for initializing resource requests, completing the initialization of the transmission module, memory mapping module, and task execution module, and registering parameter configuration files externally to support dynamic parameter configuration and querying.

[0055] The transmission module provides interfaces for creating and destroying control plane connections, supporting the establishment of control plane connections with remote nodes using the TCP protocol to exchange remote memory information. It also provides interfaces for creating and destroying data plane connections, supporting the establishment of data plane connections with remote nodes using the RDMA transmission protocol. Through the control plane connection, it exchanges information such as QPN (Queue Pair Number), GID (Global Identifier), QP (Queue Pair), and PSN (Packet Sequence Number) to achieve out-of-band connection establishment of QP. It provides remote load and remote store interfaces, supports reading data from remote nodes using RDMA read requests, and supports writing modified data from local nodes back to remote nodes using RDMA write requests.

[0056] The memory mapping module provides memory mapping and demapping interfaces, supporting the mapping of memory information from a remote node to a virtual character device on the local node. It manages the remote memory block, i.e., the target memory data, in units of pages. The page size can be set according to user needs, with a default page size of 4KB (the default value is used in the following descriptions). User-mode application processes can operate on the virtual character device through the standard system call interfaces open, mmap, or close, mapping the virtual character device to user space. When a user-mode process reads memory information, it notifies the memory mapping module by triggering a read command. The kernel mapping module requests a physical page (target memory data) on the local node, calls the remote load interface provided by the transport module, reads the target memory data corresponding to the memory information onto the local node, establishes a page table mapping, and completes the RDMA read request processing. Furthermore, the memory mapping module manages all page sizes (pages) of the target memory device. Write protection is set on the TableEntry (page table entry). When a user-mode process modifies the target memory data, it will notify the memory mapping module in the form of triggering a page fault exception. The memory mapping module adds a dirty flag to the corresponding physical page, periodically checks the dirty flag of the physical page, and writes the modified physical page back to the remote node by calling the remote store interface of the transport module.

[0057] The task execution module, transmission module, and memory mapping module encapsulate various asynchronous events into a task and register it in the task execution module. The task execution module calls the task processing interface in a loop to trigger task execution. The task types mainly include connection establishment, message reception, data plane poll CQ, page timeout check, and dirty page identification.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a remote memory read / write method based on the RDMA transfer protocol is provided, which is applied to... Figure 1 Taking a remote memory aggregation read / write system within a local node as an example, the methods for connecting a local node to a remote node include:

[0059] S202: Based on the memory information transmitted from the remote node, create a virtual character device and map the memory information to the virtual character device.

[0060] Specifically, after the local node and each remote node have completed resource initialization, the remote memory aggregation read / write system (hereinafter referred to as the system) receives the memory information transmitted by the remote node, sets the aggregation mapping strategy according to the user's business requirements, and creates a virtual character device based on the aggregation mapping strategy and the memory information, wherein the memory information has been mapped to the virtual character device.

[0061] For example, the memory information includes memory name, memory length, starting address, and rkey, etc., which are not limited in the embodiments of this application.

[0062] S204: When a page fault is triggered, physical memory is allocated and an RDMA read request is initiated to the remote node. The RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in physical memory.

[0063] Specifically, the local node can be used to run application processes. When the application process needs to read target memory data, if the virtual character device does not cache the required target memory data, a page fault will be triggered. The local node allocates physical memory and initiates an RDMA read request to the remote node. The remote node returns the target memory data corresponding to the memory information. The system establishes a page table mapping, stores the target memory data in physical memory, and outputs the target memory data to the application process.

[0064] S206, set write protection for the target memory data. When the target memory data is modified by a user-space process, mark the target memory data with a dirty flag. Perform periodic checks on the target memory data at preset time intervals. When the target memory data is detected to carry a dirty flag, initiate an RDMA write request to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty flag to the remote node.

[0065] The preset time interval can be set according to the actual situation, and is not limited in this embodiment.

[0066] Specifically, the system implements PTE write protection for the target memory data. When a user-mode application process needs to modify the target memory data, the system detects the modification and marks the target memory data as dirty. The system periodically checks whether the target memory data carries a dirty mark at preset time intervals. When it detects target memory data with a dirty mark, it outputs an RDMA write request, outputting the modified target memory data to the remote node, thus completing data synchronization between the remote node and the local node.

[0067] S208 reclaims physical memory when a physical memory timeout is detected.

[0068] Specifically, the system periodically checks the target memory data in physical memory. If any target memory data in physical memory remains unused for an extended period, the physical memory is deemed to have timed out, and the system reclaims the physical memory, thus improving the efficiency of physical memory usage.

[0069] In the aforementioned remote memory read / write method based on the RDMA transmission protocol, by supporting users to customize the aggregation mapping strategy according to their needs, the memory information of each remote node is aggregated and mapped to a virtual device within the local node. When a page fault is triggered, an RDMA read request is initiated to read the target memory data of the remote node into the local node in a timely manner. When it is detected that the target memory data has been modified, an RDMA write request is initiated to write the modified target memory data back to the remote node, thereby accelerating the data synchronization performance between the local node and the remote node. Compared with traditional technologies where not all acquired data is necessary, this application supports on-demand caching of target memory data, improving memory utilization efficiency.

[0070] In one embodiment, a virtual character device is created based on memory information transmitted from a remote node, including:

[0071] Initiate a TCP connection establishment request to the remote node. The TCP connection establishment request is used to instruct the remote node and the local node to establish a TCP connection.

[0072] A virtual character device is created based on memory information.

[0073] Specifically, after the local node and remote node have completed initialization, the system uses the TCP protocol to establish a control plane link with the remote node to exchange memory information. The system outputs a TCP connection establishment request to the remote node, which receives the TCP connection establishment request, completes the establishment of a TCP connection with the local node, and outputs memory information to the local node. The local node creates a virtual device on the local node according to the memory information name, where the virtual device corresponds to the memory information of the corresponding remote node. The system creates a virtual character device according to the target memory data required by the application process, i.e., the business requirement information, and the virtual device.

[0074] It should be noted that since the application process may need to use target memory information in different remote nodes, virtual character devices are created based on business requirements and corresponding virtual devices to support on-demand caching of target memory data, thereby improving memory utilization efficiency.

[0075] To facilitate understanding by those skilled in the art, the application scenario of a remote memory read / write method is illustrated using the process of aggregating and mapping the memory of two remote nodes from a local node as an example. Figure 3 As shown, the local node includes multiple application processes (application process 1, application process 2, ..., application process N, where N is a positive integer greater than or equal to 2). Both the local and remote nodes have systems configured. The system within the local node requests physical pages (target memory data). Remote node 1 and remote node 2 respectively read the pages (target memory data) corresponding to the memory information onto the local node. Figure 3RDMAREAD / WRITE in the code corresponds to an RDMA read request or an RDMA write request; the application process calls the system through the open interface, mmap interface, or close interface.

[0076] For example, the steps for creating a virtual device on a local node are illustrated with a specific example, such as... Figure 4 As shown:

[0077] The local node, remote node 1, and remote node 2 complete the initialization of module resources and scan the IB device. Remote node 1 completes the registration of the memory region (MR).

[0078] The local node initiates a kernel TCP connection establishment request to the remote node 1, and the remote node 1 accepts the kernel TCP connection establishment request.

[0079] Remote node 1 sends available memory information to the local node, including memory name, memory length, starting address, and rkey. The remote node 1 sends the memory name as ram1 and the memory length as 64MB. After receiving the memory information, the local node creates a virtual device rls_ram1 on the local node based on the memory name ram1.

[0080] The local node repeats the above steps to establish a kernel TCP connection with the remote node 2, and creates virtual devices rls_ram2 and rls_ram3 on the local node based on the memory names (ram2 and ram3) in the memory information of the remote node 2.

[0081] Local node application process 1 sets the aggregation mapping strategy "mem_agg1=rls_ram1, rls_ram2" according to business requirements information, and creates a virtual character device mem_agg1 on the local node;

[0082] Local node application process 2 sets the aggregation mapping strategy "mem_agg2=rls_ram1, rls_ram3" according to business requirements information, and creates a virtual character device mem_agg2 on the local node.

[0083] In this embodiment, after establishing a TCP connection with the remote node, a virtual character device is created based on memory information. A virtual device is also created based on business requirements and the virtual character device, supporting subsequent on-demand caching of target memory data and reducing the usage of local physical memory. At the same time, it supports flexible setting of aggregation strategies, which can map discrete physical memory from different remote nodes to local virtual contiguous memory, improving the flexibility of memory mapping.

[0084] In one embodiment, when a page fault is triggered, physical memory is allocated, and an RDMA read request is initiated to the remote node, including:

[0085] Supports user-space calls to the open and mmap interfaces;

[0086] Initiate an RDMA QP negotiation message to the remote node. The RDMA QP negotiation message is used to instruct the remote node and the local node to establish a QP connection.

[0087] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, if it is determined that the virtual character device has not cached the target memory data, trigger a page fault and initiate an RDMA read request.

[0088] Allocate physical memory and cache the target memory data of the remote node in physical memory.

[0089] Specifically, in user space, the `open` interface is called to open the virtual character device. The system checks whether a QP connection has been established between the local node and the remote node. If not, the creation process begins. The local node creates an RDMA QP negotiation message and sends it to the remote node via kernel TCP. After receiving the RDMA QP negotiation message returned by the remote node, a QP connection is established with the remote node. The user space maps the target memory data of the virtual character device by calling the `mmap` interface, based on the target memory data required by the application process. It obtains the length and offset of the target memory data in the remote node. Based on the length and offset, it determines whether the virtual character device caches the target memory data. If it is determined that the virtual device does not cache the target memory data, a page fault is triggered, and the system initiates an RDMA read request. The remote node returns the target memory data corresponding to the memory information. The system establishes a page table mapping, caches the target memory data from the remote node in the allocated physical memory, and outputs the target memory data to the application process.

[0090] In this embodiment, when a QP connection is established between the local node and the remote node, a page fault is triggered when it is determined that the virtual device has not cached the target memory data, based on the length and offset of the target memory data in the remote node. That is, a page fault is triggered during the process of the user process mapping the remote memory, and then physical memory is allocated on the local node, the target memory data is read into the local physical memory, and a page table mapping is established with the virtual address of the user process. This realizes on-demand caching of remote memory according to user needs, thereby improving memory utilization efficiency.

[0091] In one embodiment, the virtual character device supports the open interface and the mmap interface; the method further includes:

[0092] The virtual character device is opened by calling the open interface;

[0093] The target memory data is mapped to user space by calling the mmap interface.

[0094] Specifically, the local application process uses the standard system call open and mmap interfaces. Without needing to be aware of data communication with remote nodes or the allocation and reclamation of local memory resources, it can read and write data in remote memory as if it were accessing local memory, thus improving the data synchronization performance between nodes.

[0095] To facilitate understanding by those skilled in the art, the following example illustrates the steps of outputting an RDMA read request in response to a read command. Figure 5 As shown, the local node aggregates and maps to two remote nodes (remote node 1 and remote node 2):

[0096] Application process 1 calls the system through the open interface to open the virtual character device mem_agg1;

[0097] The local node's system rls_over_rdma checks whether an RDMA QP has been created with remote node 1 and remote node 2. If not, it enters the creation process. The local node creates an RDMA QP negotiation message and sends the RDMA QP negotiation message to remote node 1 and remote node 2 in sequence through kernelTCP. After receiving the RDMA QP negotiation message, it establishes a QP connection with remote node 1 and remote node 2.

[0098] Application process 1 calls the system through the open interface and the system returns fd 1.

[0099] Application process 1 calls the system through the mmap interface to map the memory of the virtual character device mem_agg1. The memory length is 64KB and the memory offset is 0 (the length and offset of the target memory data in the remote node).

[0100] Within the local node, the system rls_over_rdma checks whether all pages under the VMA (Virtual Memory Address) have been cached based on the memory length and memory offset. If not cached, it returns directly.

[0101] Application process 1 calls the system through the mmap interface and returns virtual address va 1.

[0102] Application process 1 reads the contents of va 1, triggering a page fault.

[0103] The local node's system rls_over_rdma requests physical memory, page_idx is 0, and initiates an RDMA read request to the page of ram1 sub_page_idx 0 on the remote node 1.

[0104] Upon receiving page 0 from remote node 1, the RDMA read request returns successfully, page table mapping is established, the reference count of page 0 is incremented by 1, a page fault is returned, and application process 1 reads the target memory data.

[0105] Application process 1 calls the system through the close interface to close the virtual character device mem_agg1, ​​the local node system rls_over_rdma closes the vma, and the reference count of page 0 is decremented by 1.

[0106] Application process 2 calls the system through the open interface to open the virtual character device mem_agg2.

[0107] The local node's system rls_over_rdma checks whether QP connections have been created between remote node 1 and remote node 2. If they have been created, it returns directly.

[0108] Application process 2 calls the system through the open interface and returns fd 2.

[0109] Application process 2 calls the system through the mmap interface to map the memory information of the virtual device mem_agg2, with a memory length of 64KB and a memory offset of 64M.

[0110] Within the local node, the system rls_over_rdma checks whether all pages under this vma have been cached based on the memory length and memory offset. If they have been cached, it returns directly.

[0111] Application process 2 calls the system through the mmap interface and returns virtual address va 2.

[0112] Application process 2 reads the contents of va 2, triggering a page fault.

[0113] The local node's system rls_over_rdma requests physical memory with page_idx of 16384, and initiates an RDMA read request to page 3 sub_page_idx of 16384 on the remote node 2.

[0114] Upon receiving page 16384 from remote node 2, the RDMA read request returns successfully, page table mapping is established, the reference count of page 16384 is incremented by 1, a page fault is returned, and application process 2 reads the target memory data.

[0115] Application process 2 calls the system through the close interface to close the virtual character device mem_agg2, disable the VMA in the local node system rls_over_rdma, and decrement the reference count of page 16384 by 1.

[0116] In one embodiment, after initiating an RDMA write request to the remote node when the target memory data is detected to carry a dirty flag, the following steps are included:

[0117] Clear the dirty flag and reset write protection for the target memory data.

[0118] Specifically, when the system finishes writing back the target memory data carrying the dirty flag to the remote node, the system clears the dirty flag carried by the target memory data, resets write protection for the target memory data, and repeats the steps of periodically checking the target memory data at preset time intervals.

[0119] In this embodiment of the application, by writing back the target memory data carrying the dirty flag to the remote node, the dirty flag of the target memory data is initially removed, and write protection is reset for the target memory data, thus ensuring the data synchronization performance between the local node and the remote node.

[0120] To facilitate understanding by those skilled in the art, the following example illustrates the steps of initiating an RDMA write request to a remote node when a target memory data is detected to carry a dirty flag. Figure 6 As shown, the local node aggregates and maps to two remote nodes (remote node 1 and remote node 2):

[0121] Application process 1 calls the system through the open interface to open the virtual device mem_agg1.

[0122] The local node's system rls_over_rdma checks whether an RDMA QP connection has been created between the local node and the remote node 1. If it has been created, it returns directly.

[0123] Application process 1 calls the system through the open interface, and the system returns fd 3.

[0124] Application process 1 calls the system through the mmap interface to map the memory information of the virtual character device mem_agg1, ​​with a memory length of 64KB and a memory offset of 0.

[0125] Within the local node, the system rls_over_rdma checks whether page 0 (the target memory data) under this VMA has been cached based on the memory length and memory offset, establishes a page table mapping, and increments the reference count of page 0 by 1.

[0126] Application process 1 calls the system through the mmap interface, and the system returns the virtual address va 3.

[0127] Application process 1 modifies the contents of VA 3. Because the local node's system rls_over_rdma is set with PTE write protection, a page fault exception is triggered, and page 0 is marked as a dirty page (marked with a dirty flag).

[0128] The local node's system rls_over_rdma periodically checks the dirty page flags of all pages, initiates an RDMA write request to page 0 of ram1 sub_page_idx 0 on remote node 1, returns the dirty page page 0 to remote node 1, and the RDMA write request returns successfully.

[0129] Application process 1 calls the system through the close interface to close the virtual device mem_agg1, ​​the local node system rls_over_rdma closes the vma, and the reference count of page 0 is decremented by 1.

[0130] Application process 2 calls the system through the open interface to open the virtual character device mem_agg2;

[0131] The local node's system rls_over_rdma checks whether an RDMA QP connection has been created between the local node and the remote node 2. If it has been created, it returns directly.

[0132] Application process 2 calls the system through the open interface and the system returns fd 4.

[0133] Application process 2 calls the system return via the mmap interface, mapping the memory information of the virtual device mem_agg2, with a memory length of 64KB and a memory offset of 64M.

[0134] Within the local node, the system rls_over_rdma checks whether page 16384 (the target memory data) under this VMA has been cached based on the memory length and memory offset, establishes a page table mapping, and increments the reference count of page 16384 by 1.

[0135] Application process 2 calls the system through the mmap interface and the system returns virtual address va 4.

[0136] Application process 2 modifies the contents of VA 4. Because the local node's system rls_over_rdma is set with PTE write protection, a page fault exception is triggered, and page 16384 is marked as a dirty page.

[0137] The local node's system rls_over_rdma periodically checks the dirty page flags of all pages, initiates an RDMA write request to page 16384 of ram3 sub_page_idx in remote node 2, returns the dirty page 16384 to remote node 1, and the RDMA write request returns successfully.

[0138] Application process 2 calls the system through the close interface to close the virtual device mem_agg2, and the system rls_over_rdma within the local node closes the vma, decrementing the reference count of page 16384 by 1.

[0139] In one embodiment, when a physical memory timeout is detected, physical memory is reclaimed, including:

[0140] Supports user-space calls to the close and munmap interfaces;

[0141] Set the physical memory to an idle state and record the idle time;

[0142] Periodically check the status of physical memory, and reclaim physical memory when it is idle for a preset time.

[0143] Destroy the QP connection between the remote node and the local node.

[0144] The time interval for periodically checking the state of physical memory can be set according to actual conditions, and is not limited in this embodiment.

[0145] The preset duration can be set according to the actual situation, and is not limited in this embodiment.

[0146] Specifically, the user space calls the munmap interface to release the mapping between memory information and the virtual character device, and calls the close interface to close the virtual character device. If it is determined that the virtual device has cached the target memory data and the target memory data does not carry a dirty flag, the system will not initiate an RDMA read request or RDMA write request, sets the physical memory to an idle state, and records the current idle time. The system periodically checks the state of the physical memory. When the physical memory is in an idle state for a preset duration, it confirms the clearing of the target memory data, reclaims the physical memory, and destroys the QP connection between the remote node and the local node after clearing all target memory data corresponding to the memory information. Otherwise, it re-checks the state of the physical memory.

[0147] In this embodiment, by periodically detecting the state of physical memory, when physical memory is detected to be idle for a preset duration, the target memory data is cleared, physical memory is reclaimed, and after clearing all target memory data corresponding to memory information, the QP connection between the remote node and the local node is destroyed. This achieves the periodic reclamation of idle local physical memory and improves the utilization efficiency of local physical memory.

[0148] In one embodiment, the virtual character device supports the close interface and the munmap interface, and the method includes:

[0149] By calling the munmap interface, the mapping relationship between memory information and virtual character devices is removed;

[0150] The virtual character device is closed by calling the close interface.

[0151] Specifically, the local application process uses the standard system calls munmap and close to read and write data to remote memory as if it were accessing local memory, without needing to be aware of data communication with remote nodes or the allocation and reclamation of local memory resources, thus improving the data synchronization performance between nodes.

[0152] To facilitate understanding by those skilled in the art, the steps for detecting target memory data according to a second preset time interval are explained below with reference to a specific example, such as... Figure 7 As shown, the local node aggregates and maps to two remote nodes (remote node 1 and remote node 2):

[0153] The local node's system rls_over_rdma periodically checks all cached physical pages (cached target memory data) within the local node.

[0154] Check if the page's reference count is 0. If it is not 0 and is still in use, return to the above steps of periodically checking all cached physical pages in the local node. Otherwise, check if the page has entered the idle state (mark it with the second tag). If it has not entered the idle state, mark the page as idle and add a timestamp, then return to the above steps of periodically checking all cached physical pages in the local node.

[0155] Based on the timestamp and the user-defined timeout threshold, check if the page has timed out. If it has not timed out, return to the steps described above for periodically checking all cached physical pages within the local node. Otherwise, clear the timed-out page, reclaim physical memory, and check if all pages under the corresponding remote node have been reclaimed. If not, return to the steps described above for periodically checking all cached physical pages within the local node. Otherwise, destroy the RDMA QP connection with the remote node and return to the steps described above for periodically checking all cached physical pages within the local node.

[0156] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0157] Based on the same inventive concept, this application also provides a remote memory read / write device for implementing the aforementioned remote memory read / write method based on the RDMA transmission protocol. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the remote memory read / write device based on the RDMA transmission protocol provided below can be found in the limitations of the remote memory read / write method described above, and will not be repeated here.

[0158] In one exemplary embodiment, such as Figure 8 As shown, a remote memory read / write device 800 based on the RDMA transmission protocol is provided. The device 800 is applied to a local node, which connects to a remote node. The device 800 includes:

[0159] The device creation module 801 is used to create a virtual character device based on the memory information transmitted from the remote node and to map the memory information to the virtual character device.

[0160] The read module 802 is used to allocate physical memory and initiate an RDMA read request to the remote node when a page fault is triggered; wherein, the RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in physical memory;

[0161] The write module 803 is used to set write protection for target memory data. When the target memory data is modified by a user-space process, the target memory data is marked with a dirty flag. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry a dirty flag, an RDMA write request is initiated to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty flag to the remote node.

[0162] The memory management module 804 is used to reclaim physical memory when a physical memory timeout is detected.

[0163] In one embodiment, the device creation module 801 is further configured to initiate a TCP connection establishment request to the remote node, the TCP connection establishment request being used to instruct the remote node and the local node to establish a TCP connection.

[0164] Based on memory information, a virtual character device is created, supporting user-mode calls to the open and mmap interfaces.

[0165] In one embodiment, the reading module 802 is further configured to initiate an RDMAQP negotiation message to the remote node, the RDMAQP negotiation message being used to instruct the remote node and the local node to establish a QP connection.

[0166] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, if it is determined that the virtual character device has not cached the target memory data, trigger a page fault and initiate an RDMA read request.

[0167] Allocate physical memory and cache the target memory data of the remote node in physical memory.

[0168] In one embodiment, the write module 803 is also configured to clear the dirty flag and reset write protection for the target memory data.

[0169] In one embodiment, the memory management module 804 is also configured to support user-mode calls to the close interface and the munmap interface;

[0170] When no RDMA read or write request is initiated, the physical memory is set to an idle state, and the idle time is recorded.

[0171] Periodically check the status of physical memory, and reclaim physical memory when it is idle for a preset time.

[0172] Destroy the QP connection between the remote node and the local node.

[0173] In one embodiment, the virtual character device supports the open interface and the mmap interface; the device creation module 801 is also used to open the virtual character device by calling the open interface;

[0174] The target memory data is mapped to user space by calling the mmap interface.

[0175] In one embodiment, the virtual character device supports the close interface and the munmap interface. The memory management module 804 is also used to release the mapping relationship between memory information and the virtual character device by calling the munmap interface.

[0176] The virtual character device is closed by calling the close interface.

[0177] The modules in the aforementioned remote memory read / write device based on the RDMA transmission protocol can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of the computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0178] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a remote memory read / write method based on the RDMA transmission protocol. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0179] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned remote memory read / write method based on the RDMA transfer protocol.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the aforementioned remote memory read / write method based on the RDMA transfer protocol.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned remote memory read / write method based on the RDMA transfer protocol.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A remote memory read / write method based on the RDMA transmission protocol, characterized in that, The method is applied to a local node, which is used to connect to a remote node; the method includes: Based on the memory information transmitted by the remote node, a virtual character device is created, and the memory information is mapped to the virtual character device; When a page fault is triggered, physical memory is allocated, and an RDMA read request is initiated to the remote node; wherein, the RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in the physical memory; Write protection is set for the target memory data. When the target memory data is modified by a user-mode process, the target memory data is marked with a dirty tag. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry the dirty tag, an RDMA write request is initiated to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty tag to the remote node. When the physical memory times out, the physical memory is reclaimed.

2. The method according to claim 1, characterized in that, The creation of a virtual character device based on the memory information transmitted by the remote node includes: A TCP connection establishment request is initiated to the remote node, the TCP connection establishment request being used to instruct the remote node and the local node to establish a TCP connection; Based on the memory information, the virtual character device is created.

3. The method according to claim 1, characterized in that, When a page fault is triggered, allocating physical memory and initiating an RDMA read request to the remote node includes: Supports user-space calls to the open and mmap interfaces; An RDMA QP negotiation message is initiated to the remote node, the RDMA QP negotiation message being used to instruct the remote node and the local node to establish a QP connection; The length and offset of the target memory data in the remote node are obtained. Based on the length and the offset, when it is determined that the virtual character device has not cached the target memory data, the page fault is triggered and the RDMA read request is initiated. Allocate the physical memory and cache the target memory data of the remote node in the physical memory.

4. The method according to claim 3, characterized in that, After the step of initiating an RDMA write request to the remote node when the target memory data is detected to carry the dirty flag, the following steps are included: Clear the dirty flag and reset write protection for the target memory data.

5. The method according to claim 1, characterized in that, When a timeout is detected in the physical memory, the physical memory is reclaimed, including: Supports user-mode calls to the close and munmap interfaces; Set the physical memory to an idle state and record the idle time; The status of the physical memory is periodically checked, and when the physical memory is in an idle state for a preset time, the physical memory is reclaimed. Destroy the QP connection between the remote node and the local node.

6. The method according to claim 3, characterized in that, The virtual character device supports the open interface and the mmap interface; the method further includes: The virtual character device is opened by calling the open interface; By calling the mmap interface, the target memory data is mapped to the user space.

7. The method according to claim 5, characterized in that, The virtual character device supports the close interface and the munmap interface, and the method includes: By calling the munmap interface, the mapping relationship between the memory information and the virtual character device is released; The virtual character device is closed by calling the close interface.

8. A remote memory read / write device based on the RDMA transmission protocol, characterized in that, The device is applied to a local node, which is used to connect to a remote node; the device includes: The device creation module is used to create a virtual character device based on the memory information transmitted by the remote node, and to map the memory information to the virtual character device; The read module is used to allocate physical memory and initiate an RDMA read request to the remote node when a page fault is triggered; wherein, the RDMA read request is used to read target memory data corresponding to the memory information from the remote node and store it in the physical memory; The write module is used to set write protection for the target memory data. When the target memory data is modified by a user-space process, the target memory data is marked with a dirty flag. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry the dirty flag, an RDMA write request is initiated to the remote node. The RDMA write request is used to write back the target memory data carrying the dirty flag to the remote node. The memory management module is used to reclaim the physical memory when a timeout is detected.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images