Method and system for sharing RDMA (Remote Direct Memory Access) equipment in Linux command space

Abstract

The invention discloses a method and a system for sharing RDMA (Remote Direct Memory Access) equipment in Linux command spaces, and the method comprises the following steps: adding a new partial sharing mode for an RDMA mode for sharing the RDMA equipment between the Linux command spaces on the basis of an original sharing mode and an exclusive mode, linux command space record data structures of the RDMA devices are modified to support that one RDMA device is simultaneously associated with a plurality of specified Linux command spaces, and operations of adding the RDMA device into the Linux command spaces and exiting the Linux command spaces are modified to be compatible with a sharing mode, an exclusive mode and a newly added partial sharing mode. According to the method, the partial sharing mode is newly added, and the Linux command space record data structure of the RDMA equipment is modified into the list, so that the problem that only specific RDMA equipment cannot be appointed to be used when the sharing mode is used is solved, and meanwhile, the problem that the number of supported containers is limited in an exclusive mode is solved.

Description

Technical Field

[0001] The present invention belongs to the field of communication and computer technology, and in particular relates to a method and system for sharing an RDMA device in a Linux command space. Background Art

[0002] Remote Direct Memory Access (RDMA) networks have become the mainstream of data center networks in recent years due to their excellent characteristics of extremely low latency, low CPU overhead and high bandwidth. Its types include InfiniBand networks and RoCE (Remote Direct Memory Access over Ethernet) networks. RDMA supports data transfer directly from the memory of one computer to the memory of another computer without the intervention of the operating systems of both parties.

[0003] The Linux namespace provides a kernel-level method to isolate system resources. It achieves resource isolation by placing the system's global resources in different namespaces. Programs in different namespaces can enjoy independent system resources. Currently, Linux provides six types of system resource isolation mechanisms, namely file system mount point isolation, host name and domain name information isolation, inter-process communication isolation, process ID isolation, network resource isolation (Ethernet), and user and user group ID isolation. There is no native implementation of RDMA device isolation in the current Linux system.

[0004] The InfiniBand driver implements two RDMA modes in Namespace, namely shared mode and exclusive mode. When the RDMA subsystem is in shared mode, the RDMA device is accessible in all Namespaces; when RDMA device isolation between multiple network namespaces is not required, the shared mode can be used. When the RDMA subsystem is in exclusive mode, the RDMA device can only be accessed in one Namespace and is not visible in other Namespaces. Currently, the industry is using these two modes in container scenarios to support the use of RDMA devices in multiple containers, but both modes have certain problems. The shared mode can be used normally when a server has only one RDMA device, but on a server with multiple RDMA devices at the same time, all RDMA devices are visible in all Namespaces. In some scenarios (such as multi-NUMA or multi-socket), container users need to use specific RDMA devices to obtain the best performance, so they need to manually specify them according to the situation. Otherwise, the first discovered device will be used by default, resulting in traffic congestion, uneven use, or the use of cross-NUMA or socket RDMA devices, resulting in reduced performance. Exclusive mode can associate only the best RDMA to a specific Namespace for container use, and other RDMA devices are invisible in the Namespace. However, RDMA devices associated with a specific Namespace also become invisible in other Namespaces. The number of RDMA devices is limited. Even if multiple VFs (virtual functions) are virtualized when SR-IOV (single root input and output virtualization) is turned on, only a limited number of containers can be started on a server, which reduces the deployment density of containers and causes resource waste. Some configuration changes of SR-IOV require modifying the device firmware and restarting the server device. Therefore, when the mode needs to be modified due to scenario requirements, it may be necessary to restart the server to make the configuration effective, and the operation process is complicated. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for sharing RDMA devices in a Linux command space are provided, which solves the problem that it is impossible to specify to use only a specific RDMA device when using the shared mode, and solves the problem that the number of containers supported in the exclusive mode is limited.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for sharing an RDMA device in a Linux command space comprises the following steps: adding a new partial sharing mode to an RDMA mode for sharing an RDMA device between Linux command spaces on the basis of an original sharing mode and an exclusive mode, modifying a Linux command space record data structure of an RDMA device so that it supports an RDMA device to be associated with multiple designated Linux command spaces at the same time, and modifying operations of an RDMA device joining a Linux command space and exiting a Linux command space so that it is compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode.

[0007] Furthermore, modifying the Linux command space record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux command spaces at the same time means modifying the Linux command space record data structure of the RDMA device into a list, and listing the specified Linux command space information to which the RDMA device is simultaneously associated through the list, so that the RDMA device supports being associated with multiple Linux command spaces at the same time, and each Linux command space can only discover and use the RDMA device recorded in the list according to the list.

[0008] Further, after modifying the operation of adding the RDMA device to the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode and the newly added partial shared mode, the operation of adding the RDMA device to the Linux command space includes: S101, generating a network link request to modify the Linux command space of the RDMA device; S102, sending a network connection request to the Linux kernel driver; S103, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces. If the RDMA mode is a shared mode, an error code is returned, the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partially shared mode, the process jumps to step S105; S104, stop the RDMA device, modify the Linux command space record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the original Linux command space, and ends and exits; S105, modify the Linux command space record data structure of the RDMA device, keep the original Linux command space unchanged, add a new Linux command space, end and exit.

[0009] Furthermore, the network link request generated in step S101 includes a setting command CMD_SET of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux command space.

[0010] Further, after modifying the operation of adding the RDMA device to the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode and the newly added partial shared mode, the operation of the RDMA device exiting the Linux command space includes: S201, generating a network link request to delete the RDMA device from a Linux command space; S202, sending a network connection request to the Linux kernel driver; S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces, and returns an error code if it is a shared mode or an exclusive mode, ends and exits, and jumps to step S204 if it is a partial shared mode; S204, determine whether the requested Linux command space exists in the Linux command space record data structure of the RDMA device. If it does not exist in the Linux command space record data structure of the RDMA device, return an error code, end and exit, otherwise, delete the requested Linux command space from the Linux command space record data structure of the RDMA device, end and exit.

[0011] Furthermore, the network connection request generated in step S201 includes a delete command CMD_DEL of the network connection request, an index INDEX of the RDMA device corresponding to the network connection request, and a file descriptor fd of the Linux command space.

[0012] Furthermore, the method further comprises: S301, pre-allocate and configure the corresponding RDMA device for each Linux command space, and record it in the initial value of the Linux command space record data structure of each RDMA device; S302, after each Linux command space is started, discovering and using the RDMA devices that can be used by the Linux command space according to the content of the Linux command space record data structure of each RDMA device, so that each RDMA device is only used by the Linux command space recorded in its Linux command space record data structure; S303, during the operation of each Linux command space, if it is necessary to modify the operation of the RDMA device to join the Linux command space or exit the Linux command space, the operation of modifying the RDMA device to join the Linux command space or exit the Linux command space is executed to update the Linux command space record data structure of the RDMA device and jump to step S303.

[0013] The present invention also proposes a system for sharing RDMA devices in a Linux command space, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for sharing RDMA devices in the Linux command space.

[0014] The present invention also proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the method of sharing the RDMA device in the Linux command space through a processor.

[0015] The present invention also proposes a computer program product, comprising a computer program / instruction, wherein the computer program / instruction is programmed or configured to execute the method for sharing an RDMA device in a Linux command space through a processor.

[0016] Compared with the prior art, the advantages of the present invention are: The present invention modifies the Linux command space record data structure of the RDMA device into a list, so that it supports one RDMA device to be associated with multiple specified Linux command spaces at the same time. For one Linux command space, all RDMA devices associated with this Linux command space can be used. For one RDMA device, it can be associated with multiple Linux command spaces at the same time, thereby solving the problem that the original sharing mode cannot specify to use only specific RDMA devices, and solving the problem that the number of containers supported by the exclusive mode is limited. In addition, the present invention also adds a new partial sharing mode on the basis of the original sharing mode and exclusive mode, and modifies the operation of the RDMA device joining the Linux command space and exiting the Linux command space to make it compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode, without switching the mode in different scenarios, thereby solving the problem that the device may need to be restarted when changing the mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0018] Figure 2 This is a flow chart of adding an RDMA device to a namespace in an embodiment of the present invention.

[0019] Figure 3 This is a flowchart of an RDMA device exiting a namespace in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Embodiment 1 This embodiment proposes a method for sharing RDMA devices in a Linux command space, which can share RDMA devices in a specified Linux command space list. For a Linux command space, all RDMA devices associated with this Linux command space can be used. For an RDMA device, it can be associated with multiple Linux command spaces at the same time, solving the problem that the original sharing mode cannot specify the use of only specific RDMA devices, and solving the problem that the number of containers supported by the exclusive mode is limited. Figure 1 As shown, the following steps are included: For the RDMA mode of sharing RDMA devices between Linux command spaces, a new partial sharing mode (partshared) is added on the basis of the original sharing mode and exclusive mode, and the Linux command space record data structure of the RDMA device is modified to support an RDMA device to be associated with multiple specified Linux command spaces at the same time, and the operations of RDMA devices joining and exiting the Linux command space are modified to make it compatible with the sharing mode, exclusive mode and the newly added partial sharing mode.

[0021] In this embodiment, modifying the Linux command space record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux command spaces at the same time means modifying the Linux command space record data structure of the RDMA device into a list, and listing the specified Linux command space information to which the RDMA device is simultaneously associated through the list, so that the RDMA device supports being associated with multiple Linux command spaces at the same time, and each Linux command space can only discover and use the RDMA device recorded in the list according to the list.

[0022] Through the above steps, this embodiment adds a third mode, the partial sharing mode partshared, on the basis of the original shared mode shared and exclusive mode exclusive. The new partial sharing mode supports sharing RDMA devices in a list of specified Linux command spaces namespaces, that is, supports one RDMA device to be associated with multiple Linux command spaces namespaces at the same time.

[0023] For the original shared mode shared, since all RDMA devices are shared among all Linux command namespaces, there is no need for the device to join and exit the Linux command namespace process. For the original exclusive mode exclusive, joining a Linux command namespace will automatically exit the original Linux command namespace, and only the device needs to join the Linux command namespace process. Therefore, in this embodiment, when modifying the operations of RDMA devices joining the Linux command space and exiting the Linux command space to make it compatible with the shared mode, exclusive mode and the newly added partial sharing mode, the main thing is to modify the original RDMA device joining the Linux command space namespace process, increase support for the partial sharing mode partshared, and add the RDMA device exiting the Linux command space namespace process.

[0024] Specifically, in this embodiment, after modifying the operations of the RDMA device joining the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device joining the Linux command space are as follows: Figure 2 As shown, including: S101, generating a network link request for modifying the Linux command space of the RDMA device. In this embodiment, the generated network link request includes a setting command CMD_SET of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux command space; S102, sending a network connection request to the Linux kernel driver; S103, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces. If the RDMA mode is a shared mode, an error code -EOPNOTSUPP is returned, and the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partially shared mode, the process jumps to step S105; S104, stop the RDMA device, modify the Linux command space record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the original Linux command space, and ends and exits; S105, modify the Linux command space record data structure of the RDMA device, keep the original Linux command space unchanged, add a new Linux command space, end and exit.

[0025] In this embodiment, after modifying the operations of the RDMA device joining the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device exiting the Linux command space are as follows: Figure 3 As shown, including: S201, generating a network link request to delete the RDMA device from a Linux command space. In this embodiment, the generated network link request includes a deletion command CMD_DEL of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux command space; S202, sending a network connection request to the Linux kernel driver; S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces, and returns an error code if it is a shared mode or an exclusive mode, ends and exits, and jumps to step S204 if it is a partial shared mode; S204, determine whether the requested Linux command space exists in the Linux command space record data structure of the RDMA device. If it does not exist in the Linux command space record data structure of the RDMA device, that is, it does not exist in the list, return an error code, end and exit, otherwise, delete the requested Linux command space from the Linux command space record data structure of the RDMA device, that is, if it exists in the list, delete the requested Linux command space from the list, end and exit.

[0026] Embodiment 2 This embodiment provides a method for sharing an RDMA device in a Linux command space, which specifically includes the following steps: S301, pre-allocate and configure the corresponding RDMA device for each Linux command space, and record it in the initial value of the Linux command space record data structure of each RDMA device; S302, after each Linux command space is started, discovering and using the RDMA devices that can be used by the Linux command space according to the content of the Linux command space record data structure of each RDMA device, so that each RDMA device is only used by the Linux command space recorded in its Linux command space record data structure; S303, during the operation of each Linux command space, if it is necessary to modify the operation of the RDMA device joining the Linux command space or exiting the Linux command space, the operation of modifying the RDMA device joining the Linux command space or exiting the Linux command space in Example 1 is executed to update the Linux command space record data structure of the RDMA device and jump to step S303.

[0027] Embodiment 3 This embodiment proposes a system for sharing RDMA devices in a Linux command space, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for sharing RDMA devices in a Linux command space described in the first embodiment.

[0028] This embodiment also proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the method for sharing an RDMA device in the Linux command space described in Embodiment 1 through a processor.

[0029] This embodiment also proposes a computer program product, including a computer program / instruction, which is programmed or configured to execute the method for sharing an RDMA device in a Linux command space described in the first embodiment through a processor.

[0030] In summary, the present invention discloses a method and system for sharing RDMA devices in Linux command space, which solves the problem that it is impossible to specify to use only a specific RDMA device when using the sharing mode, and solves the problem that the number of containers supported in the exclusive mode is limited. Compared with the prior art, the advantages of the present invention are: (1) It not only supports sharing of RDMA devices among multiple Linux namespaces, retaining the characteristics of the sharing mode, but also supports controlling the list of shared Linux namespaces. In the scenario with multiple RDMA devices, it can support the Linux namespace to only discover and use specific RDMA devices to achieve optimal performance. (2) It supports sharing in the Linux command space namespace list specified by controlling a single RDMA device. Compared with the exclusive mode, which only supports the use of one RDMA device in one Linux command space namespace and the number of containers supported by the server depends on the number of RDMA devices and the number of virtual functions that can be virtualized by a single RDMA, it can support the running of more containers, improve the deployment density of containers and the utilization rate of servers. Moreover, compared with the exclusive mode, in which some single-root input and output virtualization sriov configuration modifications of the RDMA device need to be written into the device firmware and the server needs to be restarted, the present invention does not need to use sriov and does not have this problem.

[0031] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0032] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for sharing RDMA devices in a Linux command space, characterized in that: The invention comprises the following steps: adding a new partial sharing mode to the RDMA mode of sharing RDMA devices between Linux command spaces on the basis of the original sharing mode and exclusive mode, modifying the Linux command space record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple specified Linux command spaces at the same time, and modifying the operations of the RDMA device joining the Linux command space and exiting the Linux command space so that it is compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode.

2. The method for sharing RDMA devices in Linux command space according to claim 1, characterized in that: The modifying of the Linux command space record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux command spaces at the same time refers to modifying the Linux command space record data structure of the RDMA device into a list, and listing the specified Linux command space information to which the RDMA device is simultaneously associated through the list, so that the RDMA device supports being associated with multiple Linux command spaces at the same time, and each Linux command space can only discover and use the RDMA device recorded in the list according to the list.

3. The method for sharing RDMA devices in Linux command space according to claim 2, characterized in that: After modifying the operations of adding the RDMA device to the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of adding the RDMA device to the Linux command space include: S101, generating a network link request to modify the Linux command space of the RDMA device; S102, sending a network connection request to the Linux kernel driver; S103, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces. If the RDMA mode is a shared mode, an error code is returned, the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partially shared mode, the process jumps to step S105; S104, stop the RDMA device, modify the Linux command space record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the original Linux command space, and ends and exits; S105, modify the Linux command space record data structure of the RDMA device, keep the original Linux command space unchanged, add a new Linux command space, end and exit.

4. The method for sharing RDMA devices in Linux command space according to claim 3, characterized in that: The network link request generated in step S101 includes a setting command CMD_SET of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux command space.

5. The method for sharing RDMA devices in Linux command space according to claim 3, characterized in that: After modifying the operations of adding the RDMA device to the Linux command space and exiting the Linux command space to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device exiting the Linux command space include: S201, generating a network link request to delete the RDMA device from a Linux command space; S202, sending a network connection request to the Linux kernel driver; S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux command spaces, and returns an error code if it is a shared mode or an exclusive mode, ends and exits, and jumps to step S204 if it is a partial shared mode; S204, determine whether the requested Linux command space exists in the Linux command space record data structure of the RDMA device. If it does not exist in the Linux command space record data structure of the RDMA device, return an error code, end and exit, otherwise, delete the requested Linux command space from the Linux command space record data structure of the RDMA device, end and exit.

6. The method for sharing RDMA devices in Linux command space according to claim 5, characterized in that: The network link request generated in step S201 includes a delete command CMD_DEL of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux command space.

7. The method for sharing RDMA devices in Linux command space according to claim 6, characterized in that: include: S301, pre-allocate and configure the corresponding RDMA device for each Linux command space, and record it in the initial value of the Linux command space record data structure of each RDMA device; S302, after each Linux command space is started, discovering and using the RDMA devices that can be used by the Linux command space according to the content of the Linux command space record data structure of each RDMA device, so that each RDMA device is only used by the Linux command space recorded in its Linux command space record data structure; S303, during the operation of each Linux command space, if it is necessary to modify the operation of the RDMA device to join the Linux command space or exit the Linux command space, the operation of modifying the RDMA device to join the Linux command space or exit the Linux command space is executed to update the Linux command space record data structure of the RDMA device and jump to step S303.

8. A system for sharing RDMA devices in a Linux command space, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the method for sharing RDMA devices in the Linux command space as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the method for sharing RDMA devices in the Linux command space described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program / instructions, characterized in that The computer program / instruction is programmed or configured to execute the method for sharing RDMA devices in the Linux command space described in any one of claims 1 to 7 through a processor.

Images