Rdma-based memory usage method, system, electronic device and storage medium
By obtaining the physical address and binding the physical page after receiving a message in RDMA communication, the problem of wasted physical memory in RDMA communication is solved, and memory utilization is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN JAGUAR MICROSYSTEMS CO LTD
- Filing Date
- 2022-09-13
- Publication Date
- 2026-07-31
AI Technical Summary
In RDMA communication, requesting and locking memory regions in advance may result in wasted physical memory space, as the peer RDMA device may not access all locked physical memory space.
By receiving and parsing the messages from the other end, the virtual address is determined, and the target physical address is obtained from the physical address pool. After writing the data, command completion information is written into memory to bind the physical page, thus avoiding premature locking of the physical page.
It improves the utilization of physical memory space, achieves memory over-sale effect, and avoids the waste of physical memory space.
Smart Images
Figure CN115470156B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network communication technology, and in particular to a memory usage method, system, electronic device, and storage medium based on RDMA. Background Technology
[0002] RDMA (Remote Direct Memory Access) technology was developed to address the latency issues in server-side data processing during network transmission. RDMA devices directly transfer data to the computer's storage area over the network, enabling the rapid movement of data from one system to a remote system's memory without affecting the operating system or requiring significant computer processing power.
[0003] During RDMA communication, one end of the computer needs to pre-register a memory region in memory for use by the other end's RDMA device. This requires pre-allocating and locking (i.e., pinning) the memory region, that is, allocating physical pages in memory to store data sent by the other end's RDMA device. However, the pre-allocated and locked memory region may occupy a large amount of physical memory space, and the other end's RDMA device may not access all the locked physical memory space during actual communication, thus easily leading to a waste of physical memory space. Summary of the Invention
[0004] Therefore, it is necessary to provide an RDMA-based memory usage method, system, electronic device, computer-readable storage medium, and computer program product that can improve memory space utilization in response to the above-mentioned technical problems.
[0005] Firstly, this application provides a memory usage method based on RDMA. The method includes:
[0006] The first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory area pre-registered by the local computer device for the second RDMA device; the first RDMA device is communicatively connected to the local computer device;
[0007] The first RDMA device obtains the target physical address from the physical address pool; the physical address pool is pre-established in the first RDMA device by the local computer device; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device.
[0008] The first RDMA device writes the target data into the target physical page pointed to by the target physical address, and writes command completion information into the memory of the local computer device;
[0009] The command completion information is used to instruct the local computer device to bind the target physical address to the virtual address in order to lock the target physical page.
[0010] Secondly, this application also provides a memory usage system based on RDMA. The system includes: a first RDMA device and a local computer device; the first RDMA device is communicatively connected to the local computer device; wherein:
[0011] The first RDMA device is used to receive and parse the message sent by the second RDMA device at the other end, obtain the target data to be written, and determine the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory region pre-registered by the local computer device for the second RDMA device;
[0012] The first RDMA device is further configured to obtain a target physical address from a physical address pool; the physical address pool is pre-established in the first RDMA device by the local computer device; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device.
[0013] The first RDMA device is further configured to write the target data into the target physical page pointed to by the target physical address, and write command completion information into the memory of the local computer device; the local computer device is configured to bind the target physical address to the virtual address in response to the command completion information, so as to lock the target physical page.
[0014] In one embodiment, the local computer device is further configured to pre-register the memory region for the second RDMA device to obtain the virtual address range corresponding to the memory region registered for the second RDMA device; and to record information of unlocked memory in the physical buffer list corresponding to each virtual address within the virtual address range.
[0015] In one embodiment, the local computer device is further configured to, upon reading the command completion information, record the binding relationship between the target physical address and the virtual address in a physical buffer list to lock the target physical page.
[0016] In one embodiment, if the virtual address is already bound to a physical address in the operating system of the local computer device, the local computer device is further configured to copy the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound when the command completion information is read.
[0017] In one embodiment, the message is an RDMA SEND message; the local computer device is further configured to generate a first work queue element based on a virtual address belonging to the virtual address range, and write the first work queue element into a receive queue; the first work queue element is used to instruct the first RDMA device to receive the message sent by the second RDMA device;
[0018] The first RDMA device receives and parses the RDMA SEND message sent by the second RDMA device at the peer end to obtain the target data to be written; it parses the first work queue element to obtain the virtual address.
[0019] In one embodiment, the message is a write operation message; the first RDMA device is further configured to receive and parse the message sent by the peer's second RDMA device, obtain the target data to be written, determine the virtual address to which the target data is to be written, and obtain the virtual address from the write operation message.
[0020] In one embodiment, the message is a read response message; the local computer device is further configured to generate a second work queue element based on a virtual address belonging to the virtual address range, and write the second work queue element into a send queue; the second work queue element is configured to instruct the first RDMA device to actively request the second RDMA device to read data from the memory of the peer computer device;
[0021] The first RDMA device is further configured to read and parse the second work queue element from the transmit queue to obtain the virtual address, and send a data read request to the second RDMA device; receive and parse the read response message returned by the peer's second RDMA device in response to the data read request to obtain the target data to be written; and read the virtual address obtained from the parsed second work queue element.
[0022] Thirdly, this application also provides an electronic device. The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the RDMA-based memory usage method described in various embodiments of this application.
[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, causes the processor to perform the steps of the RDMA-based memory usage method described in the embodiments of this application.
[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the steps of the RDMA-based memory usage method described in the embodiments of this application.
[0025] The aforementioned RDMA-based memory usage method, system, electronic device, storage medium, and computer program product involve a first RDMA device receiving and parsing a message sent by a second RDMA device at the other end, obtaining the target data to be written, determining the virtual address to which the target data will be written, retrieving the target physical address from the physical address pool, writing the target data into the target physical page pointed to by the target physical address, and writing command completion information into the memory of the local computer device. Upon reading the command completion information, the local computer device binds the target physical address to the virtual address to lock the target physical page. Therefore, the local computer device does not need to pre-allocate a physical address pointing to a specific physical page for the second RDMA device at the other end; instead, it only needs to register a virtual memory space, i.e., allocate a range of virtual addresses. This avoids pre-locking specific physical pages for the second RDMA device at the other end, and instead obtains the specific target physical page for data writing only after receiving the message, thereby improving the utilization rate of physical memory space. Attached Figure Description
[0026] Figure 1 This is an application environment diagram of an RDMA-based memory usage method in one embodiment;
[0027] Figure 2 This is a flowchart illustrating a memory usage method based on RDMA in one embodiment;
[0028] Figure 3 This is a schematic diagram of the overall process of a memory usage method based on RDMA in one embodiment;
[0029] Figure 4 This is a block diagram of a memory usage system based on RDMA in one embodiment;
[0030] Figure 5 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.
[0032] The RDMA-based memory usage method provided in this application can be applied to, for example... Figure 1 In the application environment shown, the first RDMA device 102 at the local end is communicatively connected to the first computer device 104 at the local end, and the second RDMA device 106 at the remote end is communicatively connected to the second computer device 108 at the remote end. The first RDMA device 102 and the second RDMA device 106 communicate via a network. The first RDMA device 102 can access data in the memory of the second computer device 108 through the second RDMA device 106, and the second RDMA device 106 can also access data in the memory of the first computer device 104 through the first RDMA device 102.
[0033] Specifically, the local first computer device 104 can pre-establish a physical address pool in the first RDMA device 102. The first computer device 104 can pre-register memory regions for the remote second RDMA device 106, obtaining the virtual address range corresponding to the registered memory regions for the second RDMA device. The first RDMA device 102 can receive and parse messages sent by the remote second RDMA device 106, obtain the target data to be written, and determine the virtual address to which the target data is to be written. The first RDMA device 102 can obtain the target physical address from the physical address pool, write the target data into the target physical page pointed to by the target physical address, and write command completion information into the memory of the local first computer device 104. Upon reading the command completion information, the first computer device 104 can bind the target physical address to the virtual address to lock the target physical page.
[0034] The first RDMA device 102 and the second RDMA device 106 can be devices based on RDMA technology. The first RDMA device 102 and the second RDMA device 106 can be respectively housed in a DPU (Data Processing Unit) chip, which can communicate with the corresponding computer device. The first computer device 104 and the second computer device 108 can be terminals or servers. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Servers can be implemented using independent servers or server clusters composed of multiple servers.
[0035] In one embodiment, such as Figure 2 As shown, an RDMA-based memory usage method is provided, including the following steps:
[0036] Step 202: The first RDMA device receives and parses the message sent by the second RDMA device at the other end, obtains the target data to be written, and determines the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory area pre-registered by the local computer device for the second RDMA device; the first RDMA device is communicatively connected to the local computer device.
[0037] RDMA devices are devices based on Remote Direct Memory Access (RDMA) technology. A virtual address is a virtual address that does not point to a specific physical page. The target data to be written refers to the payload of the message. It can be understood that a message is generated by adding auxiliary information such as data size and checksum to the original data; the original data in the message excluding the auxiliary information is the payload. A memory region is a virtual memory area corresponding to a virtual address range.
[0038] Specifically, the local computer equipment (i.e., Figure 1 The first computer device 104 can pre-register a memory region for the second RDMA device at the other end, obtaining the virtual address range corresponding to the memory region registered for the second RDMA device. The first RDMA device can receive and parse messages sent by the second RDMA device at the other end, obtain the target data to be written, and determine the virtual address to which the target data should be written. The virtual address belongs to the virtual address range corresponding to the memory region pre-registered by the local computer device for the second RDMA device.
[0039] In one embodiment, the determined virtual address can be obtained by parsing from a message. In another embodiment, the determined virtual address can be obtained by parsing from a work queue element issued by the local computer device.
[0040] In one embodiment, after the local computer device pre-registers memory regions for the second RDMA device at the other end and obtains the virtual address range corresponding to the registered memory regions for the second RDMA device, the local computer device can record information about unlocked memory corresponding to each virtual address within the virtual address range. After the first RDMA device receives and parses the message sent by the second RDMA device at the other end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, the first RDMA device can, in response to the information about unlocked memory corresponding to the virtual address, execute the steps of obtaining the target physical address from the physical address pool and subsequent steps.
[0041] Step 204: The first RDMA device obtains the target physical address from the physical address pool; the physical address pool is pre-established in the first RDMA device by the local computer device; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device.
[0042] The physical address pool stores multiple physical addresses. A physical page is a page in memory used to store data.
[0043] Specifically, the local computer device can pre-establish a physical address pool containing physical addresses in the first RDMA device. The first RDMA device can obtain physical addresses from the physical address pool as target physical addresses.
[0044] In one embodiment, when the physical addresses in the physical address pool are about to be exhausted, the local computer device can fill the physical address pool with new physical addresses.
[0045] Step 206: The first RDMA device writes the target data into the target physical page pointed to by the target physical address, and writes command completion information into the memory of the local computer device.
[0046] The command completion information instructs the local computer device to bind the target physical address to the virtual address to lock the target physical page. This command completion information indicates that the first RDMA device has written the target data into the target physical page pointed to by the target physical address.
[0047] Specifically, the first RDMA device can write the target data into the target physical page pointed to by the target physical address, and then write command completion information into the memory of the local computer device. Upon reading the command completion information, the local computer device can bind the target physical address to the virtual address to lock the target physical page.
[0048] In one embodiment, the first RDMA device may write command completion information to the memory of the local computer device through, but not limited to, an event queue.
[0049] In one embodiment, after the local computer device binds the target physical address to the virtual address to lock the target physical page, if the first RDMA device receives a new message from the second RDMA device at the other end, the first RDMA device can parse the message to obtain the target data in the new message, determine the virtual address to which the target data in the new message should be written, then determine the target physical address bound to the virtual address, and then write the target data in the new message to the target physical page pointed to by the target physical address. It can be understood that after binding the target physical address to the virtual address, it indicates that the target physical address has been locked. Therefore, after receiving a new message from the second RDMA device, the first RDMA device can directly write the target data in the new message to the target physical page without having to re-obtain and bind the physical address.
[0050] In the aforementioned RDMA-based memory usage method, the first RDMA device receives and parses the message sent by the second RDMA device at the other end, obtains the target data to be written, determines the virtual address to which the target data is to be written, obtains the target physical address from the physical address pool, writes the target data into the target physical page pointed to by the target physical address, and writes command completion information into the memory of the local computer device. Upon reading the command completion information, the local computer device binds the target physical address with the virtual address to lock the target physical page. Thus, the local computer device does not need to pre-allocate a physical address pointing to a specific physical page for the second RDMA device at the other end, but only needs to register a virtual memory space, that is, allocate a range of virtual addresses. This avoids locking the specific physical page for the second RDMA device at the other end in advance, and instead obtains the specific target physical page for data writing only after receiving the message, thereby improving the utilization rate of physical memory space. Furthermore, because physical pages do not need to be locked in advance, the virtual memory space allocated by computer devices can exceed the actual physical memory space, achieving the effect of memory over-sale. That is, computer devices can allocate more virtual addresses than the actual physical addresses, and only obtain physical addresses when data needs to be written into memory, thereby further improving the utilization rate of physical memory space.
[0051] In one embodiment, before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end, the method further includes: the local computer device pre-registering a memory region for the second RDMA device to obtain a virtual address range corresponding to the memory region registered for the second RDMA device; and the local computer device recording information about unlocked memory in a physical buffer list corresponding to each virtual address within the virtual address range.
[0052] The Physical Buffer List (PBL) is a list used to record the binding relationship between virtual addresses and physical addresses.
[0053] Specifically, the local computer device can pre-register memory regions for the second RDMA device, obtaining the virtual address range corresponding to the registered memory regions for the second RDMA device. Then, it records the information of unlocked memory corresponding to each virtual address within the virtual address range in the physical buffer list. After the first RDMA device receives and parses the message sent by the second RDMA device at the other end, obtains the target data to be written, and determines the virtual address to which the target data should be written, the first RDMA device can read the information of unlocked memory corresponding to the determined virtual address from the physical buffer list. In response to the information of unlocked memory, the first RDMA device executes the steps of obtaining the target physical address from the physical address pool and subsequent steps.
[0054] It is understood that before writing the target data into memory, the first RDMA device will read the physical address bound to the virtual address from the physical buffer list according to the virtual address. Since in the embodiments of this application, the local computer device does not allocate physical addresses in advance, but only allocates a range of virtual addresses, the first RDMA device reads the information of unlocked memory corresponding to the virtual address from the physical buffer list. Thus, the first RDMA device can respond to the information of unlocked memory and execute the process of obtaining the target physical address from the physical address pool and subsequent steps.
[0055] It is understandable that the local computer device registers a range of virtual addresses for the second RDMA device but does not allocate physical addresses. Therefore, after establishing the physical buffer list, there is no need to record the binding relationship between virtual addresses and physical addresses in it. Instead, the binding relationship between virtual addresses and physical addresses is recorded in the physical buffer list only when the local computer device binds the target physical address to the virtual address.
[0056] In one embodiment, when the local computer device registers the memory region for the second RDMA device in advance, it can establish a list of physical buffers corresponding to the registered memory region.
[0057] In one embodiment, the first RDMA device can determine the memory region corresponding to the virtual address, look up the physical buffer list corresponding to the memory region, and then read the information of the unlocked memory corresponding to the virtual address from the physical buffer list.
[0058] In one embodiment, when the local computer device registers the memory region for the second RDMA device in advance, it can also establish a memory protection translation table corresponding to the registered memory region.
[0059] The Memory Protection Translation (MPT) table is a list used to record the attributes of memory regions.
[0060] In one embodiment, the attributes of a memory region may include at least one of the following: the size of the memory region, whether the memory region allows reading data, or whether the memory region allows writing data.
[0061] In one embodiment, the memory protection translation table may also record the base address of the physical buffer list corresponding to the respective memory region. The base address of the physical buffer list is used to look up the physical buffer list.
[0062] In one embodiment, the first RDMA device can determine the index identifier (key) of the memory region corresponding to the virtual address, look up the corresponding memory protection translation table based on the index identifier of the memory region, and then look up the physical buffer list based on the base address of the physical buffer list stored in the memory protection translation table, and read the information of the unlocked memory corresponding to the virtual address from the physical buffer list.
[0063] In one embodiment, the identified virtual address and memory region index identifier can be parsed from the message. In another embodiment, the identified virtual address and memory region index identifier can be parsed from a work queue element issued by the local computer device.
[0064] It is understandable that the memory region registered by the local computer device for the second RDMA device in advance is virtual and does not point to a specific physical memory space.
[0065] In the above embodiments, the local computer device pre-registers memory regions for the second RDMA device, obtaining the virtual address range corresponding to the registered memory regions for the second RDMA device. Information about unlocked memory is recorded in the physical buffer list for each virtual address within this virtual address range, without needing to allocate specific physical addresses. This avoids the problem of pre-locking specific physical pages for the peer's second RDMA device, which might not access the locked physical pages, leading to wasted physical memory space and improving physical memory utilization. Furthermore, because physical pages don't need to be pre-locked, the virtual memory space registered by the computer device can exceed the actual physical memory space, achieving a memory over-allocation effect. That is, the computer device can allocate more virtual addresses than the actual physical addresses, only acquiring physical addresses when data needs to be written to memory, further improving physical memory utilization.
[0066] In one embodiment, the method further includes: upon reading command completion information, the local computer device records the binding relationship between the target physical address and the virtual address in a physical buffer list to lock the target physical page.
[0067] Specifically, upon receiving command completion information, the local computer device can lock the target physical page by matching the virtual address record to the target physical address in the physical buffer list.
[0068] In one embodiment, after the local computer device records the binding relationship between the target physical address and the virtual address in the physical buffer list to lock the target physical page upon receiving command completion information, if the first RDMA device receives a new message from the second RDMA device at the other end, the first RDMA device can determine the virtual address to which the target data in the new message is to be written, then search for the target physical address bound to that virtual address in the physical buffer list, and then write the target data in the new message to the target physical page pointed to by that target physical address. It can be understood that in this embodiment, because the virtual address is already bound to the target physical address, there is no need to obtain a physical address from the physical address pool; instead, the target data in the new message can be directly written to the target physical page pointed to by that target physical address.
[0069] In the above embodiments, when the local computer device reads the command completion information, it records the binding relationship between the target physical address and the virtual address in the physical buffer list to lock the target physical page. This achieves the goal of locking the specific physical page only after writing data into memory, avoiding the waste of physical memory space and improving the utilization rate of physical memory space.
[0070] In one embodiment, the method further includes: if the virtual address is already bound to a physical address in the operating system of the local computer device, then upon reading the command completion information, the local computer device copies the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound.
[0071] Specifically, after the first RDMA device writes the target data into the target physical page pointed to by the target physical address and writes command completion information into the memory of the local computer device, if the virtual address is not bound to a physical address, the local computer device can bind the target physical address to the virtual address to lock the target physical page upon reading the command completion information. If the virtual address is already bound to a physical address in the operating system of the local computer device, the local computer device can copy the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound upon upon reading the command completion information to release the target physical page.
[0072] In the above embodiments, if the virtual address is already bound to a physical address in the operating system of the local computer device, the local computer device can copy the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound, thus avoiding the virtual address being repeatedly bound to multiple physical addresses.
[0073] In one embodiment, the message is an RDMA SEND message; before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end to obtain the target data to be written and determines the virtual address to which the target data is to be written, the method further includes: the local computer device generating a first work queue element based on a virtual address within the virtual address range, and writing the first work queue element into a receive queue; the first work queue element is used to instruct the first RDMA device to receive the message sent by the second RDMA device; the first RDMA device receiving and parsing the message sent by the second RDMA device at the peer end to obtain the target data to be written and determining the virtual address to which the target data is to be written includes: the first RDMA device receiving and parsing the RDMA SEND message sent by the second RDMA device at the peer end to obtain the target data to be written; the first RDMA device parsing the first work queue element to obtain the virtual address.
[0074] The RDMA SEND message is transmitted when both ends of the computer device need to send work queue elements. A work queue element (WQE) is sent from the computer device to the RDMA device and is used to instruct the RDMA device to perform a task. The receive queue (RQ) is a type of work queue used to store work queue elements indicating received messages. The work queue (WQ) is a queue used to store work queue elements sent by the computer device. The computer device sends work queue elements to the work queue, and the RDMA device reads work queue elements from the work queue.
[0075] Specifically, the local computer device can generate a first working queue element based on a virtual address within the virtual address range, and write the first working queue element into the receive queue. The remote computer device (i.e. Figure 1 The second computer device 108 can issue a work queue element instructing the second RDMA device to send a message to the send queue of the peer computer device. The second RDMA device can read the work queue element instructing the second RDMA device to send a message from the send queue, retrieve the target data from the memory of the peer computer device according to the instruction of the work queue element, generate an RDMA SEND message, and send the RDMA SEND message to the first RDMA device. The first RDMA device receives and parses the RDMA SEND message sent by the second RDMA device, obtains the target data, reads the first work queue element from the receive queue and parses it to obtain the virtual address to which the target data is to be written, and then executes the first RDMA device to obtain the target physical address from the physical address pool and subsequent steps. The send queue (SQ) is a type of work queue used to store work queue elements instructing the sending of messages.
[0076] like Figure 3 The diagram shown illustrates the overall flow of RDMA-based memory usage methods in various embodiments of this application, assuming an RDMA SEND message scenario. Specifically, it includes the following steps:
[0077] S302, the computer device (i.e., the local computer device) pre-establishes a physical address pool in the RDMA device (i.e., the first RDMA device).
[0078] S304, the computer device pre-registers memory regions to obtain the virtual address range corresponding to the registered memory regions.
[0079] S306, The computer device sends work queue elements to the receive queue in memory.
[0080] S308, the RDMA device receives and parses the message to obtain the target data.
[0081] S310, the RDMA device reads and parses the work queue elements to obtain the virtual address and the index identifier of the memory protection translation table.
[0082] S312, the RDMA device reads the memory protection translation table according to the index identifier of the memory protection translation table, and then looks up the physical buffer list according to the base address in the memory protection translation table, and obtains the information of the unlocked memory corresponding to the virtual address from the physical buffer list.
[0083] S314, the RDMA device obtains the target physical address from the physical address pool.
[0084] S316, the RDMA device writes the target data in the message to the target physical page pointed to by the target physical address.
[0085] S318, the RDMA device writes command completion information into the memory of the computer device.
[0086] S320: When a computer device reads command completion information, it records the binding relationship between the virtual address and the target physical address in the physical buffer list.
[0087] In the above embodiments, for RDMA SEND messages, the first RDMA device obtains the target physical address from the physical address pool only after parsing the virtual address from the work queue element, without having to pre-allocate a physical address for the peer RDMA device before receiving the RDMA SEND message, thus avoiding waste of physical memory space and improving memory space utilization.
[0088] In one embodiment, the message is a write operation message; the first RDMA device receives and parses the message sent by the second RDMA device at the other end to obtain the target data to be written, and determines the virtual address to which the target data is to be written, including: the first RDMA device receives and parses the write operation message sent by the second RDMA device at the other end to obtain the target data to be written; the first RDMA device obtains the virtual address from the write operation message.
[0089] The write operation message, specifically the RDMA WRITE message, is transmitted when one RDMA device actively writes data to the memory of another computer device. In this embodiment, it can be understood that the second RDMA device at the other end actively writes data to the memory of the local computer device.
[0090] In one embodiment, the target data carried in the write operation message is data that the second RDMA device at the peer end actively requests to be written to the memory of the local computer device.
[0091] In one embodiment, the write operation message also carries a virtual address announced by the first RDMA device to the second RDMA device.
[0092] Specifically, the first RDMA device can advertise virtual addresses within the virtual address range allocated by the local computer device to the second RDMA device. The remote computer device can issue a work queue element to the send queue, instructing the remote second RDMA device to send a message. The second RDMA device can read the work queue element from the send queue, retrieve the target data from the remote computer device's memory in response to the instruction of the work queue element, generate a write operation message based on the virtual address and the target data, and send the write operation message to the first RDMA device. The first RDMA device receives and parses the write operation message sent by the remote second RDMA device, obtains the virtual address and the target data to be written, and then executes the steps of retrieving the target physical address from the physical address pool and subsequent steps.
[0093] In the above embodiments, for write operation messages, the first RDMA device obtains the target physical address from the physical address pool only after obtaining the virtual address from the write operation message, without having to pre-allocate a physical address for the peer RDMA device before receiving the write operation message, thus avoiding the waste of physical memory space and improving the utilization rate of memory space.
[0094] In one embodiment, the message is a read response message; before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end to obtain the target data to be written and determines the virtual address to which the target data is to be written, the method further includes: the local computer device generates a second work queue element based on a virtual address within the virtual address range, and writes the second work queue element into a send queue; the second work queue element is used to instruct the first RDMA device to actively request the second RDMA device to read data from the memory of the peer computer device; the first RDMA device reads and parses the second work queue element from the send queue to obtain the virtual address, and sends a data read request to the second RDMA device; the first RDMA device receiving and parses the message sent by the second RDMA device at the peer end to obtain the target data to be written, and determining the virtual address to which the target data is to be written includes: the first RDMA device receiving and parses the read response message returned by the second RDMA device at the peer end in response to the data read request to obtain the target data to be written; the first RDMA device reads the virtual address parsed from the second work queue element.
[0095] The read response message, also known as the RDMA READ message, is a message transmitted when one RDMA device actively reads data from the memory of another computer device. In this embodiment, it is understood that the scenario involves the first RDMA device actively reading data from the memory of the other computer device.
[0096] In one embodiment, the target data carried in the read response message is data that the first RDMA device actively requests to be read from the memory of the peer computer device.
[0097] Specifically, the local computer device can generate a second working queue element based on a virtual address within the virtual address range and write the second working queue element into the transmission queue. The first RDMA device can read and parse the second working queue element from the transmission queue to obtain the virtual address. In response to the second working queue element, the first RDMA device can send a data read request to the second RDMA device. In response to the data read request, the second RDMA device can retrieve the target data from the memory of the peer computer device, generate a read response message, and send the read response message to the first RDMA device. The first RDMA device receives and parses the read response message returned by the second RDMA device in response to the data read request, obtains the target data to be written, then reads the virtual address parsed from the second working queue element, and executes the steps of retrieving the target physical address from the physical address pool and subsequent steps.
[0098] In the above embodiments, for read response messages, the first RDMA device only looks up the target physical address from the physical address pool after receiving the message returned by the second RDMA device, without having to pre-allocate a physical address for the peer RDMA device before receiving the read response message, thus avoiding waste of physical memory space and improving memory space utilization.
[0099] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0100] Based on the same inventive concept, this application also provides an RDMA-based memory usage system for implementing the RDMA-based memory usage method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more RDMA-based memory usage system embodiments provided below can be found in the limitations of the RDMA-based memory usage method described above, and will not be repeated here.
[0101] In one embodiment, such as Figure 4 As shown, an RDMA-based memory usage system 400 is provided, including: a first RDMA device 402 and a local computer device 404, wherein:
[0102] The first RDMA device 402 is used to receive and parse the message sent by the second RDMA device at the other end, obtain the target data to be written, and determine the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory area pre-registered by the local computer device for the second RDMA device.
[0103] The first RDMA device 402 is also used to obtain a target physical address from a physical address pool; the physical address pool is pre-established in the first RDMA device 402 by the local computer device 404; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device 404.
[0104] The first RDMA device 402 is also used to write target data into the target physical page pointed to by the target physical address, and to write command completion information into the memory of the local computer device 404.
[0105] Local computer device 404 is used to bind the target physical address to the virtual address upon reading command completion information, thereby locking the target physical page.
[0106] In one embodiment, the local computer device 404 is further configured to pre-register the memory region for the second RDMA device to obtain the virtual address range corresponding to the memory region registered for the second RDMA device; and to record information about unlocked memory in the physical buffer list corresponding to each virtual address within the virtual address range.
[0107] In one embodiment, the local computer device 404 is further configured to record the binding relationship between the target physical address and the virtual address in the physical buffer list when command completion information is read, so as to lock the target physical page.
[0108] In one embodiment, if the virtual address is already bound to a physical address in the operating system of the local computer device, the local computer device 404 is further configured to copy the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound when the command completion information is read.
[0109] In one embodiment, the message is an RDMA SEND message; the local computer device 404 is further configured to generate a first work queue element based on a virtual address within the virtual address range, and write the first work queue element into a receive queue; the first work queue element is used to instruct the first RDMA device to receive the message sent by the second RDMA device. The first RDMA device 402 is further configured to receive and parse the RDMA SEND message sent by the second RDMA device at the peer end to obtain the target data to be written; and parse the first work queue element to obtain the virtual address.
[0110] In one embodiment, the message is a write operation message. The first RDMA device 402 is also configured to receive and parse the message sent by the peer second RDMA device, obtain the target data to be written, determine the virtual address to which the target data is to be written, and obtain the virtual address from the write operation message.
[0111] In one embodiment, the message is a read response message. The local computer device 404 is further configured to generate a second work queue element based on a virtual address within the virtual address range, and write the second work queue element into the send queue; the second work queue element is used to instruct the first RDMA device to actively request the second RDMA device to read data from the memory of the peer computer device. The first RDMA device 402 is further configured to read and parse the second work queue element from the send queue to obtain the virtual address, and send a data read request to the second RDMA device; receive and parse the read response message returned by the second RDMA device in response to the data read request to obtain the target data to be written; and read the virtual address obtained from the parsed second work queue element.
[0112] In the aforementioned RDMA-based memory usage system, the first RDMA device receives and parses the message sent by the second RDMA device at the other end, obtains the target data to be written, determines the virtual address to which the target data is to be written, retrieves the target physical address from the physical address pool, writes the target data into the target physical page pointed to by the target physical address, and writes command completion information into the memory of the local computer device. Upon reading the command completion information, the local computer device binds the target physical address with the virtual address to lock the target physical page. Thus, the local computer device does not need to pre-allocate a physical address pointing to a specific physical page for the second RDMA device at the other end, but only needs to register a virtual memory space, that is, allocate a range of virtual addresses. This avoids locking the specific physical page for the second RDMA device at the other end in advance, and instead obtains the specific target physical page for data writing only after receiving the message, thereby improving the utilization rate of physical memory space. Furthermore, because physical pages do not need to be locked in advance, the virtual memory space allocated by computer devices can exceed the actual physical memory space, achieving the effect of memory over-sale. That is, computer devices can allocate more virtual addresses than the actual physical addresses, and only obtain physical addresses when data needs to be written into memory, thereby further improving the utilization rate of physical memory space.
[0113] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an RDMA-based memory usage method.
[0114] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0115] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0117] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for RDMA-based memory usage, the method comprising: The method includes: The first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory area pre-registered by the local computer device for the second RDMA device; the first RDMA device is communicatively connected to the local computer device; The first RDMA device obtains the target physical address from the physical address pool; the physical address pool is pre-established in the first RDMA device by the local computer device; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device. The first RDMA device writes the target data into the target physical page pointed to by the target physical address, and writes command completion information into the memory of the local computer device; The command completion information is used to instruct the local computer device to bind the target physical address to the virtual address in order to lock the target physical page.
2. The method of claim 1, wherein, Before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end, the method further includes: The local computer device pre-registers the memory region for the second RDMA device to obtain the virtual address range corresponding to the memory region registered for the second RDMA device; The local computer device records information about unlocked memory in the physical buffer list corresponding to each virtual address within the virtual address range.
3. The method of claim 1, wherein, The method further includes: Upon receiving the command completion information, the local computer device records the binding relationship between the target physical address and the virtual address in the physical buffer list to lock the target physical page.
4. The method of claim 1, wherein, The method further includes: If the virtual address is already bound to a physical address in the operating system of the local computer device, then upon reading the command completion information, the local computer device copies the target data in the target physical page to the physical page pointed to by the physical address to which the virtual address is bound.
5. The method according to any one of claims 1 to 4, characterized in that, The message is an RDMA SEND message; before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, the method further includes: The local computer device generates a first work queue element based on a virtual address within the virtual address range and writes the first work queue element into a receive queue; the first work queue element is used to instruct the first RDMA device to receive a message sent by the second RDMA device. The first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, including: The first RDMA device receives and parses the RDMA SEND message sent by the second RDMA device at the other end to obtain the target data to be written; The first RDMA device parses the first work queue element to obtain the virtual address.
6. The method according to any one of claims 1 to 4, characterized in that, The message is a write operation message; The first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, including: The first RDMA device receives and parses the write operation message sent by the second RDMA device at the other end to obtain the target data to be written; The first RDMA device obtains the virtual address from the message of the write operation.
7. The method according to any one of claims 1 to 4, characterized in that, The message is a read response message; before the first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, the method further includes: The local computer device generates a second work queue element based on a virtual address within the virtual address range and writes the second work queue element into the transmission queue; the second work queue element is used to instruct the first RDMA device to actively request the second RDMA device to read data from the memory of the peer computer device; The first RDMA device reads and parses the second work queue element from the transmit queue to obtain the virtual address, and sends a data read request to the second RDMA device. The first RDMA device receives and parses the message sent by the second RDMA device at the peer end, obtains the target data to be written, and determines the virtual address to which the target data is to be written, including: The first RDMA device receives and parses the read response message returned by the second RDMA device at the peer in response to the data read request, and obtains the target data to be written; The first RDMA device reads the virtual address parsed from the second work queue element.
8. An RDMA-based memory usage system, comprising: The system includes a first RDMA device and a local computer device; the first RDMA device is communicatively connected to the local computer device; wherein: The first RDMA device is used to receive and parse the message sent by the second RDMA device at the other end, obtain the target data to be written, and determine the virtual address to which the target data is to be written; the virtual address belongs to the virtual address range corresponding to the memory region pre-registered by the local computer device for the second RDMA device; The first RDMA device is further configured to obtain a target physical address from a physical address pool; the physical address pool is pre-established in the first RDMA device by the local computer device; the physical addresses in the physical address pool are used to point to physical pages in the memory of the local computer device. The first RDMA device is further configured to write the target data into the target physical page pointed to by the target physical address, and write command completion information into the memory of the local computer device; The local computer device is configured to bind the target physical address to the virtual address in response to the command completion information, thereby locking the target physical page. 9.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.