Remote memory read-write method and device based on RDMA transmission protocol

Abstract

The invention relates to a remote memory read-write method and device based on an RDMA transmission protocol. The method comprises the following steps: creating a virtual character device based on memory information transmitted by a remote node, and mapping the memory information to a local virtual device; when missing page interruption is triggered, distributing a physical memory, initiating an RDMA read request, reading target memory data of a remote node to the physical memory, performing write protection on the target memory data, marking a dirty mark on the target memory data when a process of a user mode modifies the target memory data, and performing write protection on the target memory data; periodically detecting a local physical memory according to a preset time interval, when target memory data carrying a dirty mark is detected, initiating an RDMA write request, writing back the target memory data carrying the dirty mark to a far-end node, and when detecting that the physical memory is overtime, recovering the physical memory; by adopting the method, the data synchronization performance between the local node and the remote node can be accelerated, and the memory utilization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of high-performance network technology, and in particular to a remote memory reading and writing method and device based on the RDMA transmission protocol. Background Art

[0002] In application scenarios such as high-performance computing, distributed systems, and database systems, data sharing and synchronization between different nodes are required. For example, when using containers to provide corresponding services, multiple image data need to be pulled from the remote end during the initialization and startup of the container. When the size of the image data is large, it will take a long time to obtain the image data, and other processes or users are idle during this period. In addition, not all acquired data is necessary data. That is, there are currently problems with poor data synchronization performance between nodes and low memory usage efficiency. Summary of the invention

[0003] Based on this, it is necessary to provide a remote memory reading and writing method and device based on the RDMA transmission protocol that can speed up data synchronization performance between nodes and improve memory utilization efficiency in response to the above technical problems.

[0004] In a first aspect, the present application provides a remote memory reading and writing method based on the RDMA transmission protocol, the method is applied to a local node, and the local node is used to connect to a remote node; the method includes:

[0005] Based on the memory information transmitted by the remote node, a virtual character device is created, and the memory information is mapped to the virtual character device;

[0006] When a page fault interrupt is triggered, physical memory is allocated and an RDMA read request is initiated to the remote node; wherein the RDMA read request is used to read the target memory data corresponding to the memory information from the remote node and store it in the physical memory;

[0007] Set write protection for the target memory data. When the target memory data is modified by a user-mode process, mark the target memory data as dirty. Perform periodic detection on the target memory data at preset time intervals. When the target memory data is detected to carry a dirty mark, initiate an RDMA write request to the remote node. The RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node.

[0008] When physical memory timeout is detected, the physical memory is reclaimed.

[0009] In one embodiment, creating a virtual character device based on memory information transmitted by a remote node includes:

[0010] Initiate a TCP link establishment request to the remote node. The TCP link establishment request is used to instruct the remote node to establish a TCP connection with the local node.

[0011] Create a virtual character device based on the memory information.

[0012] In one embodiment, when a page fault interrupt is triggered, physical memory is allocated and an RDMA read request is initiated to a remote node, including:

[0013] Support user-mode calls to open and mmap interfaces;

[0014] Initiate an RDMA QP negotiation message to the remote node. The RDMA QP negotiation message is used to instruct the remote node to establish a QP connection with the local node.

[0015] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, when it is determined that the virtual character device does not cache the target memory data, trigger a page fault interrupt and initiate an RDMA read request.

[0016] Allocate physical memory and cache the target memory data of the remote node into the physical memory.

[0017] In one embodiment, when it is detected that the target memory data carries a dirty mark, after the step of initiating an RDMA write request to the remote node, the method includes:

[0018] Clear the dirty mark and re-set write protection for the target memory data.

[0019] In one embodiment, when a physical memory timeout is detected, reclaiming the physical memory includes:

[0020] Support user-mode calls to close and munmap interfaces;

[0021] Set the physical memory to idle state and record the idle time;

[0022] Periodically check the state of physical memory. When the physical memory is idle for a preset period of time, reclaim the physical memory.

[0023] Destroy the QP connection between the remote node and the local node.

[0024] In one embodiment, the virtual character device supports an open interface and an mmap interface; the method further comprises:

[0025] Open the virtual character device by calling the open interface;

[0026] By calling the mmap interface, the target memory data is mapped to user mode.

[0027] In one embodiment, the virtual character device supports a close interface and a munmap interface, and the method includes:

[0028] By calling the munmap interface, the mapping relationship between memory information and virtual character devices is released;

[0029] Close the virtual character device by calling the close interface.

[0030] In a second aspect, the present application also provides a remote memory reading and writing device based on the RDMA transmission protocol, the device is applied to a local node, and the local node is used to connect to the remote node; the device includes:

[0031] A device creation module, used for creating a virtual character device based on the memory information transmitted by the remote node, and mapping the memory information to the virtual character device;

[0032] The read module is used to allocate physical memory and initiate an RDMA read request to a remote node when a page fault interrupt is triggered; wherein the RDMA read request is used to read target memory data corresponding to the memory information from the remote node and store it in the physical memory;

[0033] The write module is used to set write protection for the target memory data. When the target memory data is modified by a user-mode process, the target memory data is marked as dirty. The target memory data is periodically checked at preset time intervals. When the target memory data is detected to carry a dirty mark, an RDMA write request is initiated to the remote node. The RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node.

[0034] The memory management module is used to reclaim the physical memory when a physical memory timeout is detected.

[0035] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0037] The above-mentioned remote memory reading and writing method and device based on the RDMA transmission protocol establishes a connection between a local node and a remote node, and creates a virtual character device based on the memory information transmitted by the remote node, and maps the memory information to the local virtual device; when a page fault interrupt is triggered, physical memory is allocated, an RDMA read request is initiated, the target memory data of the remote node is read to the physical memory, and the target memory data is write-protected; when a user-mode process modifies the target memory data, the target memory data is marked with a dirty mark, and the local physical memory is periodically checked at a preset time interval. When the target memory data carrying the dirty mark is detected, an RDMA write request is initiated to write the target memory data carrying the dirty mark to the physical memory. The target memory data is written back to the remote node, and the physical memory is recovered when a physical memory timeout is detected. The present application supports users to customize the aggregation mapping strategy according to their needs, so as to realize the aggregation mapping of the memory information of each remote node to the virtual device in the local node. When a page fault interrupt is triggered, an RDMA read request is initiated to read the target memory data of the remote node to the local node in time. When it is detected that the target memory data is modified, an RDMA write request is initiated to write the modified target memory data back to the remote node, thereby accelerating the data synchronization performance between the local node and the remote node. Compared with the traditional technology in which not all acquired data are necessary data, the present application supports on-demand caching of target memory data, thereby improving memory efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a structural block diagram of a remote memory aggregation reading and writing system in one embodiment;

[0040] Figure 2 A flowchart of a remote memory reading and writing method based on the RDMA transmission protocol in one embodiment;

[0041] Figure 3 This is an application scenario diagram of a remote memory reading and writing method based on the RDMA transmission protocol in one embodiment;

[0042] Figure 4 A schematic flow chart of steps for a local node to create a virtual character device in one embodiment;

[0043] Figure 5A flowchart of the steps of allocating physical memory and initiating an RDMA read request to a remote node when a page fault interrupt is triggered in one embodiment;

[0044] Figure 6 A flowchart of the steps of initiating an RDMA write request to a remote node when it is detected that the target memory data carries a dirty mark in one embodiment;

[0045] Figure 7 A flowchart of a step of reclaiming physical memory after a physical memory timeout is detected in one embodiment;

[0046] Figure 8 It is a structural block diagram of a remote memory reading and writing device based on the RDMA transmission protocol in one embodiment;

[0047] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0050] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0051] It can be understood that “at least one” means one or more, “plurality” means two or more, and “at least a portion of an element” means a part or all of an element.

[0052] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0053] The present application provides a remote memory reading and writing method based on the RDMA (Remote Direct Memory Access) transmission protocol, which can be applied to Figure 1 In the remote memory aggregation read-write system shown in the figure, the remote memory aggregation read-write system (RLS_OVER_RDMA) is set in the local node, and the remote memory aggregation read-write system includes the following four modules:

[0054] The main module is responsible for initializing resource application, completing the initialization of the transmission module, memory mapping module and task execution module, and registering parameter configuration files to the outside to support external dynamic parameter configuration and query.

[0055] The transmission module is used to provide control plane connection creation and destruction interfaces, support the use of TCP protocol to establish control plane connections with remote nodes, exchange remote memory information, and provide data plane connection creation and destruction interfaces. It supports the use of RDMA transmission protocol to establish data plane connections with remote nodes, and exchanges QPN (Queue Pair Number), GID (Global Identifier), QP (Queue Pair), PSN (Packet Sequence Number) and other information through the control plane connection to implement QP out-of-band chain building, provide remote load and remote store interfaces, support the use of RDMA read requests to read data from remote nodes, and support the use of RDMA write requests to synchronously write the modified data of the local node back to the remote node.

[0056] The memory mapping module provides memory mapping and demapping interfaces, supports mapping the memory information of the remote node to a virtual character device of the local node, manages the remote memory block, that is, the target memory data in units of page, and can set the page size according to user needs. The default page size is 4KB (the default value is used in the following description); supports user-mode application processes to operate the virtual character device through the standard system call interface open, mmap or close, and maps the virtual character device to the user mode. When the user-mode process reads the memory information, it will notify the memory mapping module in the form of triggering the read instruction. The kernel mapping module applies for the physical page (target memory data) on the local node, calls the remote load interface provided by the transmission module, reads the target memory data corresponding to the memory information to the local node, and completes the processing of the RDMA read request after establishing the page table mapping; further, the memory mapping module completes the processing of the RDMA read request in all pte (Page Write protection is set on the physical page (Page TableEntry). When the user-mode process modifies the target memory data, it will trigger a page fault exception to notify the memory mapping module. The memory mapping module adds a dirty mark to the corresponding physical page, periodically checks the dirty mark of the physical page, and calls the remote store interface of the transmission module to write the modified physical page back to the remote node.

[0057] In the task execution module, the transmission module and the memory mapping module, various asynchronous events are encapsulated into a task and registered in the task execution module. The task execution module cyclically calls the task processing interface to trigger task execution. Task types mainly include connection establishment, message reception, data plane poll cq, page timeout check and dirty page identification.

[0058] In an exemplary embodiment, Figure 2 As shown, a remote memory reading and writing method based on the RDMA transmission protocol is provided, and the method is applied to Figure 1 Taking the remote memory aggregation read-write system in the local node as an example, the local node is connected to the remote node, and the method includes:

[0059] S202, creating a virtual character device based on the memory information transmitted by the remote node, and mapping the memory information to the virtual character device.

[0060] Specifically, when the local node and each remote node complete resource initialization, the remote memory aggregation read-write system (hereinafter referred to as the system) receives the memory information transmitted by the remote node, and sets the aggregation mapping strategy according to the business demand information required by the user, and creates a virtual character device based on the aggregation mapping strategy and memory information, wherein the memory information has been mapped to the virtual character device.

[0061] Exemplarily, the memory information includes a memory name, a memory length, a starting address, and an rkey, etc., which are not limited in the embodiments of the present application.

[0062] S204, when a page fault interrupt is triggered, physical memory is allocated and an RDMA read request is initiated to a remote node; wherein the RDMA read request is used to read target memory data corresponding to the memory information from the remote node and store it in the physical memory.

[0063] Specifically, the local node can be used to run the application process. When the application process needs to read the target memory data, the virtual character device does not cache the required target memory data, which triggers a page fault interrupt. The local node allocates physical memory and initiates an RDMA read request to the remote node. The remote node returns the target memory data corresponding to the memory information. The system establishes a page table mapping, stores the target memory data in the physical memory, and outputs the target memory data to the application process.

[0064] S206, set write protection for the target memory data. When the target memory data is modified by a user-mode process, mark the target memory data with a dirty mark. Perform periodic detection on the target memory data at preset time intervals. When it is detected that the target memory data carries the dirty mark, initiate an RDMA write request to the remote node, wherein the RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node.

[0065] Among them, the preset time interval can be set according to actual conditions and is not limited in the embodiments of the present application.

[0066] Specifically, the system sets PTE write protection for the target memory data. When the application process in user mode needs to modify the target memory data, the system detects that the target memory data has been modified and marks the target memory data as dirty. The system periodically detects whether the target memory data carries a dirty mark at a preset time interval. When the target memory data carrying the dirty mark is detected, an RDMA write request is output to output the modified target memory data to the remote node, completing the data synchronization between the remote node and the local node.

[0067] S208, when it is detected that the physical memory has timed out, reclaim the physical memory.

[0068] Specifically, the system will periodically detect the target memory data in the physical memory. If the target memory data in any physical memory is unused for a long time, the physical memory is determined to have timed out, and the system reclaims the physical memory, thereby improving the efficiency of physical memory use.

[0069] In the above-mentioned remote memory reading and writing method based on the RDMA transmission protocol, the user is supported to customize the aggregation mapping strategy according to the needs, so as to realize the aggregation mapping of the memory information of each remote node into the virtual device in the local node. When the page fault interrupt is triggered, an RDMA read request is initiated to read the target memory data of the remote node to the local node in time. When it is detected that the target memory data is modified, an RDMA write request is initiated to write the modified target memory data back to the remote node, thereby accelerating the data synchronization performance between the local node and the remote node. Compared with the traditional technology in which not all the acquired data are necessary data, the present application supports on-demand caching of target memory data, thereby improving the memory utilization efficiency.

[0070] In one embodiment, creating a virtual character device based on memory information transmitted by a remote node includes:

[0071] Initiate a TCP link establishment request to the remote node. The TCP link establishment request is used to instruct the remote node to establish a TCP connection with the local node.

[0072] Create a virtual character device based on the memory information.

[0073] Specifically, when the local node and the remote node have completed initialization, the system uses the TCP protocol to establish a control plane link with the remote node to realize the interaction of memory information. The system outputs a TCP link establishment request to the remote node. The remote node receives the TCP link establishment request, completes the establishment of a TCP connection with the local node, and outputs the memory information to the local node; the local node creates a virtual device at the local node according to the memory information name, wherein the virtual device corresponds to the memory information of the corresponding remote node; the system creates a virtual character device according to the target memory data required by the application process, that is, the business demand information, and the virtual device.

[0074] It should be noted that, since the application process may need to use the target memory information in different remote nodes, a virtual character device is created based on the business demand information and the corresponding virtual device to support on-demand caching of target memory data, thereby improving memory efficiency.

[0075] In order to facilitate the understanding of those skilled in the art, an application scenario of a remote memory reading and writing method is described by taking the memory process of two remote nodes mapped by a local node as an example. Figure 3 As shown, the local node includes multiple application processes (application process 1, application process 2, ..., application process N, N is a positive integer greater than or equal to 2), and the local node and the remote node are both provided with a system. The system in the local node applies for a physical page (target memory data), and the remote node 1 and the remote node 2 respectively read the page (target memory data) corresponding to the memory information to the local node. Figure 3The RDMAREAD / WRITE in the RDMAREAD / WRITE corresponds to an RDMA read request or an RDMA write request; the application process calls the system through the open interface, mmap interface or close interface.

[0076] Exemplarily, the steps of creating a virtual device on a local node are described with reference to a specific example. Figure 4 As shown:

[0077] The local node, remote node 1, and remote node 2 complete module resource initialization and scan ib device. Remote node 1 completes memory mr (memory region) registration.

[0078] The local node initiates a kernel TCP link establishment request to remote node 1, and remote node 1 accepts the kernel TCP link establishment request.

[0079] Remote node 1 sends available memory information to the local node, including memory name, memory length, start address and rkey, etc. The memory name sent by remote node 1 is ram1 and the memory length is 64MB. After receiving the memory information, the local node creates a virtual device rls_ram1 on the local node according to the memory information name ram1.

[0080] The local node repeats the above steps to complete the kernel TCP link establishment with the remote node 2, and creates virtual devices rls_ram2 and rls_ram3 on the local node according to the memory names (ram2 and ram3) in the memory information of the remote node 2.

[0081] The local node application process 1 sets the aggregation mapping policy "mem_agg1=rls_ram1, rls_ram2" according to the business demand information, and creates a virtual character device mem_agg1 on the local node;

[0082] The local node application process 2 sets the aggregate mapping policy "mem_agg2=rls_ram1, rls_ram3" according to the business demand information, and creates a virtual character device mem_agg2 on the local node.

[0083] In an embodiment of the present application, by creating a virtual character device based on memory information after completing the establishment of a TCP connection with a remote node, and creating a virtual device based on business demand information and the virtual character device, it supports subsequent on-demand caching of target memory data and reduces the usage of local physical memory; at the same time, it supports flexible setting of aggregation strategies, and can map discrete physical memories from different remote nodes to local virtual continuous memory, thereby improving the flexibility of memory mapping.

[0084] In one embodiment, when a page fault interrupt is triggered, physical memory is allocated and an RDMA read request is initiated to a remote node, including:

[0085] Support user-mode calls to open and mmap interfaces;

[0086] Initiate an RDMA QP negotiation message to the remote node. The RDMA QP negotiation message is used to instruct the remote node to establish a QP connection with the local node.

[0087] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, when it is determined that the virtual character device does not cache the target memory data, trigger a page fault interrupt and initiate an RDMA read request.

[0088] Allocate physical memory and cache the target memory data of the remote node into the physical memory.

[0089] Specifically, the user state calls the open interface to open the virtual character device. The system checks whether a QP connection has been established between the local node and the remote node. If not, the creation process is entered. The local node creates an RDMA QP negotiation message and sends the RDMA QP negotiation message to the remote node through kernel TCP. After receiving the RDMA QP negotiation message returned by the remote node, the local node establishes a QP connection with the remote node. The user state maps the target memory data of the virtual character device by calling the mmap interface according to the target memory data required by the application process, obtains the length and offset of the target memory data in the remote node, and determines whether the virtual character device caches the target memory data according to the length and offset. When it is determined that the virtual device does not cache the target memory data, a page fault interrupt is triggered, and the system initiates an RDMA read request. The remote node returns the target memory data corresponding to the memory information. The system establishes a page table mapping, caches the target memory data of the remote node to the allocated physical memory, and outputs the target memory data to the application process.

[0090] In an embodiment of the present application, by establishing a QP connection between a local node and a remote node, based on the length and offset of the target memory data in the remote node, when it is determined that the virtual device has not cached the target memory data, a page fault interrupt is triggered, that is, during the process of mapping the remote memory by the user process, a page fault interrupt is triggered, and physical memory is allocated on the local node, the target memory data is read into the local physical memory, and a page table mapping is established with the user process virtual address, thereby realizing on-demand caching of the remote memory according to user needs and improving memory utilization efficiency.

[0091] In one embodiment, the virtual character device supports an open interface and an mmap interface; the method further comprises:

[0092] Open the virtual character device by calling the open interface;

[0093] By calling the mmap interface, the target memory data is mapped to user mode.

[0094] Specifically, the local application process uses the standard system call open interface and mmap interface. It does not need to perceive the data communication with the remote nodes and the allocation and recovery process of local memory resources. It can read and write remote memory data just like accessing local memory, thereby improving the data synchronization performance between nodes.

[0095] To facilitate understanding by those skilled in the art, the steps of outputting an RDMA read request in response to triggering a read instruction are described below with reference to a specific example. Figure 5 As shown, the local node aggregates and maps two remote nodes (remote node 1 and remote node 2):

[0096] Application process 1 calls the system through the open interface to open the virtual character device mem_agg1;

[0097] The system rls_over_rdma in the local node checks whether the RDMA QP between remote node 1 and remote node 2 is created. If not, the creation process begins. The local node creates an RDMA QP negotiation message and sends the RDMA QP negotiation messages to remote node 1 and remote node 2 through kernelTCP in turn. After receiving the RDMA QP negotiation message, the local node establishes a QP connection with remote node 1 and remote node 2.

[0098] Application process 1 calls the system through the open interface and the system returns fd 1.

[0099] Application process 1 calls the system through the mmap interface to map the memory of the virtual character device mem_agg1, ​​with a memory length of 64KB and a memory offset of 0 (the length and offset of the target memory data in the remote node);

[0100] The system rls_over_rdma in the local node checks whether all pages under the vma (Virtual Memory Address) have been cached based on the memory length and memory offset. If not, it returns directly.

[0101] Application process 1 calls the system through the mmap interface and returns the virtual address va 1.

[0102] Application process 1 reads the content of va 1, triggering a page fault.

[0103] The system rls_over_rdma in the local node applies for physical memory, page_idx is 0, and initiates an RDMA read request to the page with ram1 sub_page_idx of 0 on the remote node 1.

[0104] Receive page 0 returned by remote node 1, the RDMA read request is successfully returned, the page table mapping is established, the reference count of page 0 is increased by 1, the page fault is returned, and application process 1 reads the target memory data.

[0105] Application process 1 calls the system through the close interface to close the virtual character device mem_agg1. The system rls_over_rdma in the local node closes vma, and the reference count of page 0 is reduced by 1.

[0106] Application process 2 calls the system through the open interface to open the virtual character device mem_agg2.

[0107] The system rls_over_rdma in the local node checks whether the QP connection between the remote node 1 and the remote node 2 is established. If it is established, it returns directly.

[0108] Application process 2 system calls the system through the open interface and returns fd 2.

[0109] Application process 2 calls the system through the mmap interface to map the memory information of the virtual device mem_agg2. The memory length is 64KB and the memory offset is 64M.

[0110] The system rls_over_rdma in the local node checks whether all pages under the vma have been cached based on the memory length and memory offset. If they have been cached, they are returned directly.

[0111] Application process 2 calls the system return through the mmap interface and returns the virtual address va 2.

[0112] Application process 2 reads the content of va 2, triggering a page fault.

[0113] The system rls_over_rdma in the local node applies for physical memory, page_idx is 16384, and initiates an RDMA read request to the page with sub_page_idx 16384 of ram 3 on the remote node 2.

[0114] Receive page 16384 returned by remote node 2, the RDMA read request is successfully returned, the page table mapping is established, the reference count of page 16384 is increased by 1, the page fault is returned, and application process 2 reads the target memory data.

[0115] Application process 2 calls the system through the close interface to close the virtual character device mem_agg2. The system rls_over_rdma in the local node closes vma, and the reference count of page 16384 is reduced by 1.

[0116] In one embodiment, when it is detected that the target memory data carries a dirty mark, after the step of initiating an RDMA write request to the remote node, the method includes:

[0117] Clear the dirty mark and re-set write protection for the target memory data.

[0118] Specifically, when the system completes writing the target memory data carrying the dirty mark back to the remote node, the system clears the dirty mark carried by the target memory data, re-sets write protection for the target memory data, and repeats the step of periodically checking the target memory data at preset time intervals.

[0119] In an embodiment of the present application, after writing back the target memory data carrying the dirty mark to the remote node, the dirty mark of the target memory data is initially removed, and write protection is re-set for the target memory data, thereby ensuring data synchronization performance between the local node and the remote node.

[0120] To facilitate understanding by those skilled in the art, the steps of initiating an RDMA write request to a remote node when it is detected that the target memory data carries a dirty mark are described below with reference to a specific example. Figure 6 As shown, the local node aggregates and maps two remote nodes (remote node 1 and remote node 2):

[0121] Application process 1 calls the system through the open interface to open the virtual device mem_agg1.

[0122] The system rls_over_rdma in the local node checks whether the RDMA QP connection to the remote node 1 is established. If it is established, it returns directly.

[0123] Application process 1 calls the system through the open interface, and the system returns fd 3.

[0124] Application process 1 calls the system through the mmap interface to map the memory information of the virtual character device mem_agg1. The memory length is 64KB and the memory offset is 0.

[0125] The system rls_over_rdma in the local node checks that page 0 (target memory data) under the vma has been cached based on the memory length and memory offset, establishes a page table mapping, and increases the reference count of page 0 by 1.

[0126] Application process 1 calls the system through the mmap interface, and the system returns the virtual address va 3.

[0127] Application process 1 modifies the content of va 3. Because the system rls_over_rdma in the local node is set with pte write protection, a page fault exception is triggered and page 0 is marked as a dirty page (with a dirty mark).

[0128] The system rls_over_rdma in the local node periodically checks the dirty page marks of all pages, initiates an RDMA write request to the page of ram1 sub_page_idx 0 of remote node 1, returns the dirty page page 0 to remote node 1, and the RDMA write request returns success.

[0129] Application process 1 calls the system through the close interface to close the virtual device mem_agg1. The system rls_over_rdma in the local node closes vma, and the reference count of page 0 is reduced by 1.

[0130] Application process 2 calls the system through the open interface to open the virtual character device mem_agg2;

[0131] The system rls_over_rdma in the local node checks whether the RDMA QP connection to the remote node 2 is established. If it is established, it returns directly.

[0132] Application process 2 calls the system through the open interface and the system returns fd 4.

[0133] Application process 2 calls the system return through the mmap interface and maps the memory information of the virtual device mem_agg2. The memory length is 64KB and the memory offset is 64M.

[0134] The system rls_over_rdma in the local node checks that page16384 (target memory data) under the vma has been cached based on the memory length and memory offset, establishes a page table mapping, and increases the reference count of page 16384 by 1.

[0135] Application process 2 calls the system return through the mmap interface, and the system returns the virtual address va 4.

[0136] Application process 2 modifies the content of va 4. Since the local node's system rls_over_rdma is set with pte write protection, a page fault exception is triggered and page 16384 is marked as a dirty page.

[0137] The system rls_over_rdma in the local node periodically checks the dirty page marks of all pages, initiates an RDMA write request to the page of ram3 sub_page_idx 16384 of the remote node 2, and returns the dirty page page 16384 to the remote node 1. The RDMA write request returns successfully.

[0138] Application process 2 calls the system through the close interface to close the virtual device mem_agg2. The system rls_over_rdma in the local node closes vma, and the reference count of page 16384 is reduced by 1.

[0139] In one embodiment, when a physical memory timeout is detected, reclaiming the physical memory includes:

[0140] Support user-mode calls to close and munmap interfaces;

[0141] Set the physical memory to idle state and record the idle time;

[0142] Periodically check the state of physical memory. When the physical memory is idle for a preset period of time, reclaim the physical memory.

[0143] Destroy the QP connection between the remote node and the local node.

[0144] Among them, the time interval corresponding to the periodic check of the state of the physical memory can be set according to actual conditions and is not limited in the embodiments of the present application.

[0145] Among them, the preset duration can be set according to actual conditions and is not limited in the embodiments of the present application.

[0146] Specifically, the user state releases the mapping relationship between the memory information and the virtual character device by calling the munmap interface, and closes the virtual character device by calling the close interface; when it is determined that the virtual device has cached the target memory data and the target memory data does not carry a dirty mark, it can be understood that the system will not initiate an RDMA read request or an RDMA write request at this time, set the physical memory to an idle state, and record the current idle time; the system periodically checks the state of the physical memory, and when the physical memory is in an idle state for a preset period of time, confirms to clear the target memory data, reclaims the physical memory, and destroys the QP connection between the remote node and the local node when all target memory data corresponding to the memory information are cleared, otherwise the state of the physical memory is re-checked.

[0147] In an embodiment of the present application, by periodically detecting the state of the physical memory, when it is detected that the physical memory is in an idle state and continues for a preset period of time, it is confirmed to clear the target memory data, reclaim the physical memory, and after completing the clearing of all target memory data corresponding to the memory information, the QP connection between the remote node and the local node is destroyed, thereby achieving regular recovery of local idle physical memory and improving the utilization efficiency of local physical memory.

[0148] In one embodiment, the virtual character device supports a close interface and a munmap interface, and the method includes:

[0149] By calling the munmap interface, the mapping relationship between memory information and virtual character devices is released;

[0150] Close the virtual character device by calling the close interface.

[0151] Specifically, the local application process uses the standard system call munmap interface and close interface. It does not need to perceive the data communication with the remote nodes and the allocation and recovery process of local memory resources. It can read and write remote memory data just like accessing local memory, thereby improving the data synchronization performance between nodes.

[0152] To facilitate understanding by those skilled in the art, the step of detecting the target memory data according to the second preset time interval is described below with reference to a specific example. Figure 7 As shown, the local node aggregates and maps two remote nodes (remote node 1 and remote node 2):

[0153] The system rls_over_rdma in the local node periodically checks all cached physical pages in the local node (cached target memory data).

[0154] Check whether the reference count of the page is 0. If it is not 0 and is still in use, return to the above step of periodically checking all cached physical pages in the local node. Otherwise, check whether the page has entered the idle state (marked with the second label). If not, mark the page as entering the idle state and mark it with a timestamp, and return to the above step of periodically checking all cached physical pages in the local node.

[0155] According to the timestamp and the timeout threshold set by the user, check whether the page has timed out. If not, return to the above step of periodically checking all cached physical pages in the local node. Otherwise, clear the timed-out page, recycle the physical memory, and check whether all pages under the corresponding remote node have been reclaimed. If not, return to the above step of periodically checking all cached physical pages in the local node. Otherwise, destroy the RDMA QP connection with the remote node and return to the above step of periodically checking all cached physical pages in the local node.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a remote memory read-write device for implementing the remote memory read-write method based on the RDMA transmission protocol involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the remote memory read-write device based on the RDMA transmission protocol provided below can refer to the limitations of the remote memory read-write method above, and will not be repeated here.

[0158] In an exemplary embodiment, Figure 8 As shown, a remote memory reading and writing device 800 based on the RDMA transmission protocol is provided. The device 800 is applied to a local node, and the local node is connected to a remote node. The device 800 includes:

[0159] The device creation module 801 is used to create a virtual character device based on the memory information transmitted by the remote node, and map the memory information to the virtual character device;

[0160] The reading module 802 is used to allocate physical memory and initiate an RDMA read request to a remote node when a page fault interrupt is triggered; wherein the RDMA read request is used to read target memory data corresponding to the memory information from the remote node and store it in the physical memory;

[0161] The write module 803 is used to set write protection for the target memory data. When the target memory data is modified by a user-mode process, the target memory data is marked as dirty. The target memory data is periodically checked at preset time intervals. When it is detected that the target memory data carries the dirty mark, an RDMA write request is initiated to the remote node, wherein the RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node.

[0162] The memory management module 804 is used to reclaim the physical memory when a physical memory timeout is detected.

[0163] In one embodiment, the device creation module 801 is further used to initiate a TCP link establishment request to a remote node, where the TCP link establishment request is used to instruct the remote node to establish a TCP connection with the local node;

[0164] Based on the memory information, a virtual character device is created to support user-mode calls to the open interface and the mmap interface.

[0165] In one embodiment, the reading module 802 is further used to initiate an RDMAQP negotiation message to a remote node, where the RDMAQP negotiation message is used to instruct the remote node to establish a QP connection with the local node;

[0166] Obtain the length and offset of the target memory data in the remote node. Based on the length and offset, when it is determined that the virtual character device does not cache the target memory data, trigger a page fault interrupt and initiate an RDMA read request.

[0167] Allocate physical memory and cache the target memory data of the remote node into the physical memory.

[0168] In one embodiment, the write module 803 is also used to clear the dirty mark and re-set write protection for the target memory data.

[0169] In one embodiment, the memory management module 804 is also used to support user mode calling close interface and munmap interface;

[0170] When no RDMA read request or RDMA write request is initiated, the physical memory is set to an idle state and the idle time is recorded;

[0171] Periodically check the state of physical memory. When the physical memory is idle for a preset period of time, reclaim the physical memory.

[0172] Destroy the QP connection between the remote node and the local node.

[0173] In one embodiment, the virtual character device supports an open interface and an mmap interface; the device creation module 801 is further used to open the virtual character device by calling the open interface;

[0174] By calling the mmap interface, the target memory data is mapped to user mode.

[0175] In one embodiment, the virtual character device supports the close interface and the munmap interface, and the memory management module 804 is further used to release the mapping relationship between the memory information and the virtual character device by calling the munmap interface;

[0176] Close the virtual character device by calling the close interface.

[0177] Each module in the above-mentioned remote memory reading and writing device based on the RDMA transmission protocol can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0178] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a remote memory reading and writing method based on the RDMA transmission protocol is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0179] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned remote memory reading and writing method based on the RDMA transmission protocol when executing the computer program.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the remote memory reading and writing method based on the RDMA transmission protocol is implemented.

[0182] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned remote memory reading and writing method based on the RDMA transmission protocol when executed by a processor.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0184] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0185] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0186] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A remote memory reading and writing method based on RDMA transmission protocol, characterized in that: The method is applied to a local node, and the local node is used to connect to a remote node; the method comprises: Creating a virtual character device based on the memory information transmitted by the remote node, and mapping the memory information to the virtual character device; When a page fault interrupt is triggered, physical memory is allocated, and an RDMA read request is initiated to the remote node; wherein the RDMA read request is used to read target memory data corresponding to the memory information from the remote node and store it in the physical memory; Set write protection for the target memory data, and when the target memory data is modified by a user-state process, mark the target memory data with a dirty mark; perform periodic detection on the target memory data at a preset time interval, and when it is detected that the target memory data carries the dirty mark, initiate an RDMA write request to the remote node, wherein the RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node; When it is detected that the physical memory has timed out, the physical memory is reclaimed.

2. The method according to claim 1, characterized in that The step of creating a virtual character device based on the memory information transmitted by the remote node includes: Initiate a TCP link establishment request to the remote node, wherein the TCP link establishment request is used to instruct the remote node to establish a TCP connection with the local node; Based on the memory information, the virtual character device is created.

3. The method according to claim 1, characterized in that When a page fault interrupt is triggered, physical memory is allocated and an RDMA read request is initiated to the remote node, including: Support the user mode calling of open interface and mmap interface; Initiate an RDMA QP negotiation message to the remote node, where the RDMA QP negotiation message is used to instruct the remote node to establish a QP connection with the local node; Acquire the length and offset of the target memory data in the remote node, and based on the length and the offset, trigger the page fault interrupt when it is determined that the virtual character device does not cache the target memory data, and initiate the RDMA read request; The physical memory is allocated, and the target memory data of the remote node is cached in the physical memory.

4. The method according to claim 3, characterized in that After the step of initiating an RDMA write request to the remote node when it is detected that the target memory data carries the dirty mark, the method further comprises: The dirty mark is cleared, and write protection is set again for the target memory data.

5. The method according to claim 1, characterized in that When it is detected that the physical memory has timed out, reclaiming the physical memory includes: Support the user mode calling close interface and munmap interface; Setting the physical memory to an idle state and recording the idle time; Periodically checking the state of the physical memory, and when the physical memory is in the idle state for a preset period of time, reclaiming the physical memory; The QP connection between the remote node and the local node is destroyed.

6. The method according to claim 3, characterized in that The virtual character device supports the open interface and the mmap interface; the method further comprises: Open the virtual character device by calling the open interface; By calling the mmap interface, the target memory data is mapped to the user state.

7. The method according to claim 5, characterized in that The virtual character device supports the close interface and the munmap interface, and the method comprises: By calling the munmap interface, the mapping relationship between the memory information and the virtual character device is released; By calling the close interface, the virtual character device is closed.

8. A remote memory reading and writing device based on the RDMA transmission protocol, characterized in that: The device is applied to a local node, and the local node is used to connect to a remote node; the device includes: A device creation module, used to create a virtual character device based on the memory information transmitted by the remote node, and map the memory information to the virtual character device; A reading module, used for allocating physical memory and initiating an RDMA read request to the remote node when a page fault interrupt is triggered; wherein the RDMA read request is used for reading target memory data corresponding to the memory information from the remote node and storing it in the physical memory; A write module, used to set write protection for the target memory data, and when the target memory data is modified by a user-state process, mark the target memory data with a dirty mark; periodically detect the target memory data at a preset time interval, and when it is detected that the target memory data carries the dirty mark, initiate an RDMA write request to the remote node, wherein the RDMA write request is used to write the target memory data carrying the dirty mark back to the remote node; The memory management module is used to reclaim the physical memory when it is detected that the physical memory has timed out.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images