Remote direct memory access system, device and cluster

Abstract

The invention provides a remote direct memory access system, equipment and a cluster. The system comprises the following steps: a remote direct memory access device sends a first address conversion request to a host under the condition that a first physical address corresponding to a first virtual address does not exist; the host queries a first physical address corresponding to the first virtual address in a memory of the host according to the first address translation request, and returns a translation failure response to the remote direct memory access device under the condition that the first physical address corresponding to the first virtual address cannot be queried; the remote direct memory access device sends a page request to the host based on the conversion failure response; the host migrates data corresponding to the first virtual address from a hard disk of the host based on the page request and locks the data in a memory of the host, and returns a page request response; and the remote direct memory access device stores the corresponding relationship between the first virtual address and the first physical address into an address translation list based on the page request response.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a remote direct memory access system, device, and cluster. Background Art

[0002] Remote direct memory access devices can access the entire physical address space of the host, such as the code segment, data segment and other memory areas of the operating system, through remote direct memory access. In order to protect the system and achieve isolation in address space, the input / output memory management unit (IOMMU) can be used to restrict remote direct memory access devices to access only specified memory areas, and to aggregate multiple scattered DMA operations into continuous DMA operations. However, address conversion through IOMMU increases the access time of DMA operations. Summary of the invention

[0003] The present application provides a remote direct memory access system, device and cluster, which can avoid frequently loading data from a host's hard disk to the host's memory and unloading data from the memory to the hard disk due to the need for address conversion.

[0004] In a first aspect, a remote direct memory access system is provided, comprising: a remote direct memory access device, configured to query, according to a first virtual address carried in a first remote direct memory access request, whether there is a first physical address corresponding to the first virtual address in an address conversion list, replace the first virtual address in the first remote direct memory access request with the first physical address if there is a first physical address corresponding to the first virtual address, and send a first address conversion request to a host if there is no first physical address corresponding to the first virtual address, wherein the first address conversion request carries the first virtual address; a host, configured to receive the first address translation request, query an input / output memory management unit of the host for a first physical address corresponding to the first virtual address according to the first address translation request, and return a translation failure response to the remote direct memory access device if the first physical address corresponding to the first virtual address cannot be queried; A remote direct memory access device, configured to receive the conversion failure response, and send a page request to the host based on the conversion failure response, wherein the page request includes the first virtual address; A host, configured to receive the page request, wherein the operating system of the host migrates the data corresponding to the first virtual address from the hard disk of the host to the memory of the host based on the page request, and returns a page request response, wherein the page request response includes the first physical address; The remote direct memory access device is used to receive the page request response, and store the corresponding relationship between the first virtual address and the first physical address in the address conversion list based on the page request response.

[0005] In the above scheme, when the remote direct memory access device sends an address conversion request to the host, if the address is not in the memory but in the hard disk, the host will return a conversion failure response to the remote direct memory access device, and after the remote direct memory access device receives the conversion failure response sent by the host, it will send a page request to the host again, and the host's operating system migrates the data corresponding to the first virtual address from the host's hard disk to the host's memory based on the page request. The above method ensures that the page request is responded to by the host's operating system, avoiding the situation that when the remote direct memory access device sends an address conversion request to the host, if the address is not in the memory but in the hard disk, the host's driver migrates the data corresponding to the first virtual address from the host's hard disk to the host's memory through a specific function, thereby locking the data in the memory. Once the user forgets to release the data, the memory will be occupied all the time. In this application, the operating system swaps the required physical address into the physical memory and updates the IOMMU page table after receiving the request, but does not fix these physical addresses, and may still swap them out to the hard disk afterwards. In scenarios where dynamic memory access is required, such as graphics processing units (GPUs) and artificial intelligence (AI), this usage increases the flexibility of memory management. The operating system only ensures that the page exists when responding and uses memory on demand without the overhead caused by fixed physical addresses.

[0006] In some possible embodiments, the host is configured to send an address invalidation request to the remote direct memory access device, wherein the address invalidation request includes the first virtual address; The remote direct memory access device is used to delete the corresponding relationship between the first virtual address and the first physical address in the address conversion list according to the address invalidation request, and send an invalidation completion response to the host.

[0007] In some possible embodiments, the remote direct memory access device is used to send a second remote direct memory access request to the host between receiving the address invalidation request and the invalidation completion response, wherein the second remote direct memory access request includes a second virtual address.

[0008] In the above scheme, when any address in the address conversion list needs to be invalidated, the address conversion is no longer performed, but the second remote direct memory access request is directly transparently transmitted to the host, thereby avoiding the incorrect use of the invalidated address and causing errors.

[0009] In some possible embodiments, the remote direct memory access device is used to query whether there is a second physical address corresponding to the second virtual address in the address conversion list according to the second virtual address carried in the second remote direct memory access request between receiving the address invalidation request and the invalidation completion response, and send the second remote direct memory access request to the host if the second physical address corresponding to the second virtual address exists.

[0010] In the above scheme, when it is necessary to invalidate any address in the address translation list, even if another address is to be queried and the address exists in the address translation list, address conversion is no longer performed, but the second remote direct memory access request is directly transmitted to the host, thereby avoiding the incorrect use of the invalidated address and causing errors.

[0011] In some possible embodiments, the remote direct memory access device is used to combine the second remote direct memory access request and the invalidation completion response into a same message, and send the message to the host.

[0012] In the above solution, the cancellation completion response is scheduled preferentially, and the second remote direct memory access request and the cancellation completion response are combined into the same message, thereby reducing the amount of data that needs to be transmitted.

[0013] In some possible embodiments, the remote direct memory access device is used to store the first address translation request in the cache of the remote direct memory access device after sending the first address translation request to the host, query whether there is a first physical address corresponding to the first virtual address in the address translation list according to the first virtual address carried in the third remote direct memory access request, and prohibit the generation of this address translation request if there is no first physical address corresponding to the first virtual address and the first address translation request is stored in the cache of the remote direct memory access device.

[0014] In the above solution, when the first address translation request is stored in the cache of the remote direct memory access device, the first address translation request will not be sent again, thereby avoiding issuing multiple address translation requests for the same address, resulting in unnecessary bandwidth waste.

[0015] In some possible embodiments, the remote direct memory access device is used to store the page request in a first-in-first-out queue.

[0016] In some possible embodiments, the remote direct memory access device is used to store the page request in the first-in-first-out queue if the first-in-first-out queue is not full, and discard the page request if the first-in-first-out queue is full.

[0017] In the above scheme, if the FIFO queue is not full, the page request is stored in the FIFO queue; if the FIFO queue is full, the page request is discarded, thereby avoiding the communication resources of the remote direct memory access request being preempted due to an excessive number of page requests.

[0018] In a second aspect, a computing device is provided, comprising a remote direct memory access system and a storage unit, wherein the remote direct memory access system and the storage unit are capable of communicating with each other, and the remote direct memory access system is a system as described in any one of the first aspects.

[0019] According to a third aspect, a computing device cluster is provided, comprising a plurality of computing devices, at least one computing device comprising a remote direct memory access system and a storage unit, wherein the remote direct memory access system and the storage unit are capable of communicating with each other, and the remote direct memory access system is a system as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0021] Figure 1 It is a structural diagram of a remote direct memory access system provided by the present application; Figure 2 is a structural diagram of another remote direct memory access system provided by the present application; Figure 3 It is a flowchart of a remote direct memory access method provided by the present application; Figure 4 It is a structural schematic diagram of a computing device provided by this application. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0023] See also Figure 1 , Figure 1 Schematic diagram of a remote direct memory access system provided by this application. Figure 1 As shown, the remote direct memory access system of the present application includes: a host 110 and a remote direct memory access device 120 .

[0024] The remote direct memory access device 120 may be a device capable of accessing the host's memory through remote direct memory access. Remote direct memory access (RDMA) is a technology that bypasses the kernel of the remote host operating system to access data in its memory. Since it does not go through the operating system, it not only saves a lot of CPU resources, but also improves system throughput and reduces the system's network communication delay. It is particularly suitable for wide application in large-scale parallel computer clusters. RDMA can use the front-end network to transmit the data in the storage space of the storage device as a message in the form of an RDMA message to the storage space of the host, or use the front-end network to transmit the data in the storage space of the host as a message in the form of an RDMA message to the storage space of the storage device. These transmission and copying tasks are mainly performed by the network interface card of the storage device and the host (for example, an RDMA network interface card (RNIC)), and do not require processor execution, so they do not affect the performance of the host client. Currently, the hardware implementations of RDMA include Infiniband (IB), RDMA over converged Ethernet (RoCE) (including RoCEv1 and RoCEv2), and Internet wide area RDMA protocol (iWARP). Among them, Infiniband is a network designed specifically for RDMA, which guarantees reliable transmission at the hardware level. RoCE and iWARP are both Ethernet-based RDMA technologies that support corresponding Verbs interfaces. RoCE also supports hardware-level guarantees for reliable transmission. Remote direct memory access devices can include network cards, storage devices, accelerators, gateways, and so on.

[0025] The host 110 may refer to a computing device or system that can provide resources, services or run applications. The host may include a server, a desktop computer, a laptop computer, a tablet computer, a smart phone, etc. It includes a memory, a memory controller and a plurality of processing units. Among them, the memory, the memory controller and the plurality of processing units are interconnected through an on-chip bus. The processing unit may be a central processing unit (CPU) for processing instructions and data stored in the memory. The processing unit on a node can quickly access the local memory of the node through the on-chip bus, and can also access the memory of other nodes through a high-speed interconnection network. For example, the processing unit 0 of node 0 can access the memory of node 0 or the memory of node 1. Compared with accessing the memory of the node, it will take more time for the processor to access the memory of other nodes. Therefore, for a non-uniform memory access (NUMA) system, the memory in the same node as the processing unit should ideally contain the information most relevant to the processing unit. In one implementation, the processing unit may be a multi-core chip, that is, a chip containing multiple processing cores. In another implementation, the processing unit may be a chip with one processing core. In addition, in some embodiments, multiple nodes of the NUMA system are located on the same chip. Memory refers to an internal memory that directly exchanges data with a processing unit. It can read and write data at any time and at a very fast speed, and serves as a temporary data storage for an operating system or other running programs. Memory includes at least two types of memory, for example, memory can be either a random access memory or a read-only memory (ROM). For example, a random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard disks, but has a slower access speed than DRAM and is cheaper than DRAM in terms of cost. However, DRAM and SCM are only exemplary descriptions in this embodiment, and memory can also include other random access memories, such as static random access memories (SRAM).As for the read-only memory, for example, it can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. In addition, the memory can also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of a dynamic random access memory (DRAM), or a solid state disk (SSD). In practical applications, multiple memories and different types of memories can be configured in a NUMA system. This embodiment does not limit the number and type of memory. In addition, the memory can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory will not be lost. The memory with a power-saving function is called a non-volatile memory. The memory controller is used to manage and plan the data transmission from the memory to the processing unit, and can be a separate chip or integrated in the chip of the processing unit.

[0026] like Figure 2 As shown, the remote direct memory access device may include: a service module and an address conversion module, wherein the address conversion module includes a table lookup module, a cache management module, an invalidation request processing module, an address conversion information cache module and an address conversion request initiation module.

[0027] The address translation information cache module is used to store an address translation list, and one or more correspondences can be stored in the address translation list. Each correspondence is used to indicate the correspondence between a pair of virtual addresses and physical addresses. Among them, the virtual address here refers to the virtual address of the host's memory, and the physical address here refers to the physical address of the host's memory. The virtual address space of the host's memory is divided into multiple virtual pages, and the physical address space of the host's memory is also divided into multiple physical pages. Then, the mapping relationship between the virtual page and the physical page can be recorded through the page table. For example, the address translation list may include the correspondence between the first virtual address and the first physical address, the correspondence between the second virtual address and the second physical address, and so on.

[0028] The business module sends a first remote direct memory access request to the table lookup module. The first remote direct memory access request carries a first virtual address, and the first virtual address is the virtual address of the memory of the host that needs to be accessed. After the table lookup module receives the first remote direct memory access request sent by the business module, the table lookup module obtains the first virtual address from the first remote direct memory access request, and sends a table lookup request for the first virtual address to the cache management module. After receiving the table lookup request, the cache management module sends a cache read enable to the address translation information cache module, so that the address translation information cache module is in a state that can be read, and then the cache management module queries the address translation list according to the first virtual address.

[0029] If the cache management module finds that the address translation information cache module stores a first physical address corresponding to the first virtual address, then after the cache management module obtains the first physical address through querying the table, the first physical address is sent to the table lookup module through the table lookup result. After receiving the first physical address, the table lookup module replaces the first virtual address in the first remote direct memory access request with the first physical address, and then sends the replaced first remote direct memory access request to the host. After receiving the first remote direct memory access request, the host obtains the first physical address from the first remote direct memory access request and uses the first physical address to access the host's memory.

[0030] If the cache management module finds that the first physical address corresponding to the first virtual address is not stored in the address translation information cache module, the cache management module sends the first virtual address to the address translation request initiation module. The address translation request initiation module sends a first address translation request to the host, wherein the first address translation request carries the first virtual address. After receiving the first virtual address, the host can search the first virtual address to the corresponding first physical address in the memory through the input / output memory management unit (IOMMU).

[0031] If the host finds the first physical address corresponding to the first virtual address, it means that the data corresponding to the first virtual address has been moved from the hard disk to the physical memory of the host, and the host sends a successful conversion response to the address conversion request initiation module in the remote direct memory access device. The successful conversion response includes the first physical address. After receiving the successful conversion response, the address conversion request initiation module in the remote direct memory access device obtains the first physical address from the successful conversion response and sends the first physical address to the cache management module. The cache management module sends a cache write enable to the address conversion information cache module, so that the address conversion information cache module processes the write-allowed state, and then the cache management module writes the correspondence between the first virtual address and the first physical address into the address conversion information cache module.

[0032] If the host finds the first physical address corresponding to the first virtual address, it means that the data corresponding to the first virtual address has not been moved from the hard disk to the physical memory of the host, and the host sends a conversion failure response to the address conversion request initiation module in the remote direct memory access device. The address conversion request initiation module in the remote direct memory access device sends the first virtual address to the page request initiation module. The page request initiation module generates a page request according to the first virtual address, and determines whether the first-in-first-out queue is full. If the first-in-first-out queue is not full, the page request is stored in the first-in-first-out queue. If the first-in-first-out queue is full, the page request is discarded. The page request initiation module sends the page request to the host. After receiving the page request, the host migrates the data corresponding to the first virtual address from the hard disk to the memory, and determines the first physical address corresponding to the first virtual address. Then, the host sends a page request response to the page request initiation module of the remote direct memory access device. Among them, the page request response includes the first physical address. The page request initiation module of the remote direct memory access device sends the first physical address to the address conversion request initiation module. The address conversion request initiation module sends the first physical address to the cache management module. The cache management module sends a cache write enable to the address translation information cache module, so that the address translation information cache module processes the write-allowed state, and then the cache management module writes the correspondence between the first virtual address and the first physical address into the address translation information cache module.

[0033] Optionally, the host may send a cancellation request to the cancellation request processing module of the remote direct memory access device. The address cancellation request includes the first virtual address. The cancellation request processing module of the remote direct memory access device sends a cancellation request to the cache management module. After receiving the cancellation request, the cache management module sends a cache write enable to the address conversion information cache module, so that the address conversion information cache module is in a state where it can be written, and finds the corresponding relationship between the first virtual address and the first physical address in the address conversion table in the address conversion information cache module according to the first virtual address, and deletes the corresponding relationship between the first virtual address and the first physical address. Then, the cache management module sends a cancellation completion response to the table lookup module. Between the time when the cache management module receives the cancellation request and the time when the cache management module sends the cancellation completion response, the cache management module does not respond to any table lookup request received. Since when a corresponding relationship is cancelled, if the management module is allowed to query the address conversion table with each other, then the corresponding relationship that needs to be cancelled may just be queried, thereby obtaining an erroneous physical address. Therefore, between the time when the cache management module receives the invalidation request and the time when the cache management module sends the invalidation completion response, if the business module sends the second remote direct memory access request (including the second virtual address) to the table lookup module, then even if the address conversion information cache module stores the correspondence between the second virtual address and the second physical address, the table lookup module will not send a table lookup request to the cache management module, but will send the second remote direct memory access request (including the second virtual address) directly to the host. Since the invalidation completion response can be scheduled first, the table lookup module first receives the second remote direct memory access request and then receives the invalidation completion response. However, the invalidation completion response can be sent to the host first and then the second remote direct memory access request. Alternatively, the invalidation completion response and the second remote direct memory access request can be merged into the same message, and the message can be sent to the host.

[0034] Optionally, after sending the first address conversion request to the host, the address conversion request initiating module of the remote direct memory access device stores the first address conversion request in the cache of the remote direct memory access device. The service request module sends a third remote direct memory access request to the table lookup module, wherein the third remote direct memory access request carries the first virtual address. After the table lookup module receives the third remote direct memory access request sent by the service module, the table lookup module obtains the first virtual address from the third remote direct memory access request, and sends a table lookup request of the first virtual address to the cache management module. After receiving the table lookup request, the cache management module sends a cache read enable to the address conversion information cache module, so that the address conversion information cache module is in a state that can be read, and then the cache management module queries the address conversion list according to the first virtual address. Since the cache management module queries that the address conversion information cache module does not store the first physical address corresponding to the first virtual address, the cache management module sends the first virtual address to the address conversion request initiating module. The address conversion request initiating module determines that the first address conversion request in the cache is a query for the first physical address corresponding to the first virtual address, so the generation of this address conversion request is prohibited, and only the result of the first address conversion request needs to be waited.

[0035] See also Figure 3 , Figure 3 is a flowchart of a remote direct memory access method provided by this application. Figure 3 As shown, the remote direct memory access method of the present application includes: S101: Query whether there is a first physical address corresponding to the first virtual address in the address conversion list according to the first virtual address carried in the first remote direct memory access request; if there is a first physical address corresponding to the first virtual address, replace the first virtual address in the first remote direct memory access request with the first physical address; if there is no first physical address corresponding to the first virtual address, send a first address conversion request to the host.

[0036] The business module sends a first remote direct memory access request to the table lookup module. The first remote direct memory access request carries a first virtual address, and the first virtual address is the virtual address of the memory of the host that needs to be accessed. After the table lookup module receives the first remote direct memory access request sent by the business module, the table lookup module obtains the first virtual address from the first remote direct memory access request, and sends a table lookup request for the first virtual address to the cache management module. After receiving the table lookup request, the cache management module sends a cache read enable to the address translation information cache module, so that the address translation information cache module is in a state that can be read, and then the cache management module queries the address translation list according to the first virtual address.

[0037] If the cache management module finds that the address translation information cache module stores a first physical address corresponding to the first virtual address, then after the cache management module obtains the first physical address through querying the table, the first physical address is sent to the table lookup module through the table lookup result. After receiving the first physical address, the table lookup module replaces the first virtual address in the first remote direct memory access request with the first physical address, and then sends the replaced first remote direct memory access request to the host. After receiving the first remote direct memory access request, the host obtains the first physical address from the first remote direct memory access request and uses the first physical address to access the host's memory.

[0038] If the cache management module finds that the address translation information cache module does not store the first physical address corresponding to the first virtual address, the cache management module sends the first virtual address to the address translation request initiation module. The address translation request initiation module sends the first address translation request to the host, wherein the first address translation request carries the first virtual address.

[0039] S102: The host receives the first address translation request, queries the first physical address corresponding to the first virtual address in the input / output memory management unit of the host according to the first address translation request, and returns a translation failure response to the remote direct memory access device if the first physical address corresponding to the first virtual address cannot be found.

[0040] If the host finds the first physical address corresponding to the first virtual address, it means that the data corresponding to the first virtual address has been moved from the hard disk to the physical memory of the host, and the host sends a successful conversion response to the address conversion request initiation module in the remote direct memory access device. The successful conversion response includes the first physical address. After receiving the successful conversion response, the address conversion request initiation module in the remote direct memory access device obtains the first physical address from the successful conversion response and sends the first physical address to the cache management module. The cache management module sends a cache write enable to the address conversion information cache module, so that the address conversion information cache module processes the write-allowed state, and then the cache management module writes the correspondence between the first virtual address and the first physical address into the address conversion information cache module.

[0041] If the host finds the first physical address corresponding to the first virtual address, it means that the data corresponding to the first virtual address has not been moved from the hard disk to the physical memory of the host. The host sends a conversion failure response to the address conversion request initiating module in the remote direct memory access device.

[0042] S103: The remote direct memory access device receives the conversion failure response, and sends a page request to the host based on the conversion failure response.

[0043] The page request includes a first virtual address. The address translation request initiation module in the remote direct memory access device sends the first virtual address to the page request initiation module. The page request initiation module generates a page request according to the first virtual address, and determines whether the first-in-first-out queue is full. If the first-in-first-out queue is not full, the page request is stored in the first-in-first-out queue. If the first-in-first-out queue is full, the page request is discarded. The page request initiation module sends the page request to the host.

[0044] S104: The host receives the page request, migrates the data corresponding to the first virtual address from the hard disk of the host to the memory of the host based on the page request, and returns a page request response.

[0045] The page request response includes the first physical address. After receiving the page request, the host migrates the data corresponding to the first virtual address from the hard disk to the memory, and determines the first physical address corresponding to the first virtual address.

[0046] S105: The remote direct memory access device receives a page request response, and stores a correspondence between the first virtual address and the first physical address in an address translation list based on the page request response.

[0047] The page request initiation module of the remote direct memory access device sends the first physical address to the address translation request initiation module. The address translation request initiation module sends the first physical address to the cache management module. The cache management module sends a cache write enable to the address translation information cache module, so that the address translation information cache module processes the write-allowed state, and then the cache management module writes the corresponding relationship between the first virtual address and the first physical address into the address translation information cache module.

[0048] See also Figure 4 , Figure 4 4 is a schematic diagram of a computing device provided by the present application. The computing device 400 includes: one or more processing units 410 , a communication interface 420 and a storage unit 430 .

[0049] The processing unit 410, the communication interface 420 and the storage unit 430 are interconnected via a bus 440. Optionally, the computing device 400 may also include an input / output interface 450, which is connected to an input / output device for receiving parameters set by a user, etc. The computing device 400 can be used to implement some or all of the functions of the device embodiment or system embodiment in the above-mentioned embodiment of the present application; the processing unit 410 can also be used to implement some or all of the operation steps of the method embodiment in the above-mentioned embodiment of the present application. For example, the specific implementation of the various operations performed by the computing device 400 can refer to the specific details in the above-mentioned embodiment, such as the processing unit 410 is used to perform some or all of the steps in the above-mentioned method embodiment or some or all of the operations in the above-mentioned method embodiment. For another example, in the embodiment of the present application, the computing device 400 can be used to implement some or all of the functions of one or more components in the above-mentioned device embodiment, and the communication interface 420 can be specifically used for the communication functions necessary to implement the functions of these devices and components, and the processing unit 410 can be specifically used for the processing functions necessary to implement the functions of these devices and components, etc.

[0050] Figure 4 The computing device 400 may include one or more processing units 410, and the multiple processing units 410 may provide processing capabilities in a coordinated manner in a parallel connection manner, a serial connection manner, a serial-parallel connection manner, or any connection manner, or the multiple processing units 410 may constitute a processor sequence or a processor array, or the multiple processing units 410 may be divided into a main processor and an auxiliary processor, or the multiple processing units 410 may have different architectures such as a heterogeneous computing architecture. In addition, Figure 4 The computing device 400 shown in the figure, and the related structural description and functional description are exemplary and non-limiting. In some exemplary embodiments, the computing device 400 may include Figure 4 More or fewer components may be shown, or some components may be combined or separated, or may have a different arrangement of components.

[0051] The processing unit 410 may have a variety of specific implementation forms. For example, the processing unit 410 may include a processor (central processing unit, CPU), a graphic processing unit (graphic processing unit, GPU), a neural-network processing unit (neural-network processing unit, NPU), a tensor processing unit (tensor processing unit, TPU) or a data processing unit (data processing unit, DPU) and other combinations thereof, which are not specifically limited in the embodiments of the present application. The processing unit 410 may also be a single-core processor or a multi-core processor. The processing unit 410 may be a combination of a CPU and a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (application-specific integrated circuit, ASIC), a programmable logic device (programmable logic device, PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a generic array logic (generic array logic, GAL) or any combination thereof. The processing unit 410 may also be implemented using a logic device with built-in processing logic alone, such as an FPGA or a digital signal processor (digital signal processor, DSP), etc. The communication interface 420 may be a wired interface or a wireless interface for communicating with other modules or devices. The wired interface may be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface may be a cellular network interface or a wireless local area network interface, etc.

[0052] The storage unit 430 may be a non-volatile memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The storage unit 430 may also be a volatile memory, which may be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). The storage unit 430 may also be used to store program codes and data, so that the processing unit 410 can call the program codes stored in the storage unit 430 to execute some or all of the operation steps in the above method embodiment, or to execute the corresponding functions in the above device embodiment. Figure 4 Show more or fewer components, or have different component configurations.

[0053] The bus 440 may be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 440 may be divided into an address bus, a data bus, a control bus, etc. In addition to the data bus, the bus 440 may also include a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0054] The present application also provides a system, which includes multiple computing devices, and the structure of each computing device can refer to the structure of the computing device described above. The functions or operations that can be implemented by the system can refer to the specific implementation steps in the above method embodiment and / or the specific functions described in the above device embodiment, which will not be repeated here.

[0055] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one network site, computer, server or data center to another network site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or it can be a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape, etc.), an optical medium (e.g., a DVD, etc.), or a semiconductor medium (e.g., a solid-state hard disk), etc. In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims

1. A remote direct memory access system, characterized in that: include: a remote direct memory access device, configured to query, according to a first virtual address carried in a first remote direct memory access request, whether there is a first physical address corresponding to the first virtual address in an address conversion list, replace the first virtual address in the first remote direct memory access request with the first physical address if there is a first physical address corresponding to the first virtual address, and send a first address conversion request to a host if there is no first physical address corresponding to the first virtual address, wherein the first address conversion request carries the first virtual address; a host, configured to receive the first address translation request, query an input / output memory management unit of the host for a first physical address corresponding to the first virtual address according to the first address translation request, and return a translation failure response to the remote direct memory access device if the first physical address corresponding to the first virtual address cannot be queried; A remote direct memory access device, configured to receive the conversion failure response, and send a page request to the host based on the conversion failure response, wherein the page request includes the first virtual address; A host, configured to receive the page request, wherein the operating system of the host migrates the data corresponding to the first virtual address from the hard disk of the host to the memory of the host based on the page request, and returns a page request response, wherein the page request response includes the first physical address; The remote direct memory access device is used to receive the page request response, and store the corresponding relationship between the first virtual address and the first physical address in the address conversion list based on the page request response.

2. The system according to claim 1, characterized in that A host, configured to send an address invalidation request to the remote direct memory access device, wherein the address invalidation request includes the first virtual address; The remote direct memory access device is used to delete the corresponding relationship between the first virtual address and the first physical address in the address conversion list according to the address invalidation request, and send an invalidation completion response to the host.

3. The system according to claim 2, characterized in that The remote direct memory access device is used to send a second remote direct memory access request to the host between receiving the address invalidation request and the invalidation completion response, wherein the second remote direct memory access request includes a second virtual address.

4. The system according to claim 3, characterized in that A remote direct memory access device is used to query whether there is a second physical address corresponding to the second virtual address in the address conversion list according to the second virtual address carried in the second remote direct memory access request between receiving the address invalidation request and the invalidation completion response, and if the second physical address corresponding to the second virtual address exists, send the second remote direct memory access request to the host.

5. The system according to claim 4, characterized in that A remote direct memory access device is used to merge the second remote direct memory access request and the cancellation completion response into a same message queue, and send all messages in the message queue to the host.

6. The system according to claim 1, characterized in that A remote direct memory access device, configured to, after sending a first address translation request to the host, store the first address translation request in a cache of the remote direct memory access device, query whether there is a first physical address corresponding to the first virtual address in an address translation list according to a first virtual address carried in a third remote direct memory access request, and prohibit generation of this address translation request when there is no first physical address corresponding to the first virtual address and the first address translation request is stored in the cache of the remote direct memory access device.

7. The system according to any one of claims 1 to 6, characterized in that: The remote direct memory access device is used for storing the page request in a first-in-first-out queue.

8. The system according to claim 7, characterized in that The remote direct memory access device is used for storing the page request in the first-in-first-out queue when the first-in-first-out queue is not full, and discarding the page request when the first-in-first-out queue is full.

9. A computing device, characterized in that It comprises a remote direct memory access system and a storage unit, the remote direct memory access system and the storage unit can communicate with each other, and the remote direct memory access system is a system as described in any one of claims 1-8.

10. A computing device cluster, characterized in that: It comprises a plurality of computing devices, at least one computing device comprises a remote direct memory access system and a storage unit, the remote direct memory access system and the storage unit can communicate with each other, and the remote direct memory access system is a system as described in any one of claims 1-8.

Images