Method and system for sharing RDMA devices in Linux namespace

Abstract

The present invention discloses a method and system for sharing RDMA devices in a Linux namespace, the method comprising: adding a new partial sharing mode to the RDMA mode for sharing RDMA devices between Linux namespaces on the basis of the original sharing mode and exclusive mode, modifying the Linux namespace record data structure of the RDMA device to support an RDMA device to be associated with multiple specified Linux namespaces at the same time, and modifying the operation of the RDMA device joining the Linux namespace and exiting the Linux namespace to be compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode. The present invention solves the problem that it is impossible to specify to use only a specific RDMA device when using the sharing mode by adding a partial sharing mode and modifying the Linux namespace record data structure of the RDMA device to be a list, and solves the problem that the number of containers supported in the exclusive mode is limited.

Description

Technical Field

[0001] The present invention belongs to the field of communication and computer technology, and in particular relates to a method and system for sharing an RDMA device in a Linux namespace. Background Art

[0002] Remote Direct Memory Access (RDMA) networks have become the mainstream of data center networks in recent years due to their excellent characteristics of extremely low latency, low CPU overhead and high bandwidth. Its types include InfiniBand networks and RoCE (Remote Direct Memory Access over Ethernet) networks. RDMA supports data transfer directly from the memory of one computer to the memory of another computer without the intervention of the operating systems of both parties.

[0003] The Linux namespace provides a kernel-level method to isolate system resources. It achieves resource isolation by placing the system's global resources in different namespaces. Programs in different namespaces can enjoy independent system resources. Currently, Linux provides six types of system resource isolation mechanisms, namely file system mount point isolation, host name and domain name information isolation, inter-process communication isolation, process ID isolation, network resource isolation (Ethernet), and user and user group ID isolation. There is no native implementation of RDMA device isolation in the current Linux system.

[0004] The InfiniBand driver implements two RDMA modes in Namespace, namely shared mode and exclusive mode. When the RDMA subsystem is in shared mode, the RDMA device is accessible in all Namespaces; when RDMA device isolation between multiple network namespaces is not required, the shared mode can be used. When the RDMA subsystem is in exclusive mode, the RDMA device can only be accessed in one Namespace and is not visible in other Namespaces. Currently, the industry is using these two modes in container scenarios to support the use of RDMA devices in multiple containers, but both modes have certain problems. The shared mode can be used normally when a server has only one RDMA device, but on a server with multiple RDMA devices at the same time, all RDMA devices are visible in all Namespaces. In some scenarios (such as multi-NUMA or multi-socket), container users need to use specific RDMA devices to obtain the best performance, so they need to manually specify them according to the situation. Otherwise, the first discovered device will be used by default, resulting in traffic congestion, uneven use, or the use of cross-NUMA or socket RDMA devices, resulting in reduced performance. Exclusive mode can associate only the best RDMA to a specific Namespace for container use, and other RDMA devices are invisible in the Namespace. However, RDMA devices associated with a specific Namespace also become invisible in other Namespaces. The number of RDMA devices is limited. Even if multiple VFs (virtual functions) are virtualized when SR-IOV (single root input and output virtualization) is turned on, only a limited number of containers can be started on a server, which reduces the deployment density of containers and causes resource waste. Some configuration changes of SR-IOV require modifying the device firmware and restarting the server device. Therefore, when the mode needs to be modified due to scenario requirements, it may be necessary to restart the server to make the configuration effective, and the operation process is complicated. Summary of the invention

[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a method and system for sharing RDMA devices in a Linux namespace are provided, which solves the problem that it is impossible to specify to use only a specific RDMA device when using the shared mode, and solves the problem that the number of containers supported in the exclusive mode is limited.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for sharing an RDMA device in a Linux namespace comprises the following steps: adding a new partial sharing mode to an RDMA mode for sharing an RDMA device between Linux namespaces on the basis of an original sharing mode and an exclusive mode, modifying a Linux namespace record data structure of an RDMA device so that it supports an RDMA device to be associated with multiple specified Linux namespaces at the same time, and modifying operations of an RDMA device joining a Linux namespace and exiting a Linux namespace so that it is compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode.

[0008] Furthermore, modifying the Linux namespace record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux namespaces at the same time means modifying the Linux namespace record data structure of the RDMA device into a list, and listing the specified Linux namespace information to which the RDMA device is simultaneously associated through the list, so that the RDMA device supports being associated with multiple Linux namespaces at the same time, and each Linux namespace can only discover and use the RDMA device recorded in the list according to the list.

[0009] Further, after modifying the operation of adding the RDMA device to the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, the exclusive mode and the newly added partial shared mode, the operation of adding the RDMA device to the Linux namespace includes:

[0010] S101, generating a network link request for modifying the Linux namespace of the RDMA device;

[0011] S102, sending a network connection request to the Linux kernel driver;

[0012] S103, the Linux kernel driver determines the RDMA mode of the shared RDMA device between the current Linux namespaces. If the RDMA mode is a shared mode, an error code is returned, and the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partial shared mode, the process jumps to step S105;

[0013] S104, stop the RDMA device, modify the Linux namespace record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the Linux namespace originally joined, and then ends and exits;

[0014] S105, modify the Linux namespace record data structure of the RDMA device, keep the original Linux namespace unchanged, add a new Linux namespace, end and exit.

[0015] Furthermore, the network connection request generated in step S101 includes a setting command CMD_SET of the network connection request, an index INDEX of the RDMA device corresponding to the network connection request, and a file descriptor fd of the Linux namespace.

[0016] Further, after modifying the operation of adding the RDMA device to the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, the exclusive mode and the newly added partial shared mode, the operation of the RDMA device exiting the Linux namespace includes:

[0017] S201, generating a network link request to delete the RDMA device from a Linux namespace;

[0018] S202, sending a network connection request to the Linux kernel driver;

[0019] S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux namespaces. If it is a shared mode or an exclusive mode, an error code is returned, and the process ends and exits. If it is a partial shared mode, the process jumps to step S204;

[0020] S204, determine whether the requested Linux namespace exists in the Linux namespace record data structure of the RDMA device. If it does not exist in the Linux namespace record data structure of the RDMA device, return an error code, end and exit, otherwise, delete the requested Linux namespace from the Linux namespace record data structure of the RDMA device, end and exit.

[0021] Furthermore, the network connection request generated in step S201 includes a delete command CMD_DEL of the network connection request, an index INDEX of the RDMA device corresponding to the network connection request, and a file descriptor fd of the Linux namespace.

[0022] Furthermore, the method further comprises:

[0023] S301, pre-allocate and configure the corresponding RDMA device for each Linux namespace, and record it in the initial value of the Linux namespace record data structure of each RDMA device;

[0024] S302, after each Linux namespace is started, discovering and using the RDMA devices available for the Linux namespace according to the content of the Linux namespace record data structure of each RDMA device, so that each RDMA device is only used by the Linux namespace recorded in its Linux namespace record data structure;

[0025] S303, during the operation of each Linux namespace, if it is necessary to modify the operation of the RDMA device joining the Linux namespace or exiting the Linux namespace, the operation of modifying the RDMA device joining the Linux namespace or exiting the Linux namespace is executed to update the Linux namespace record data structure of the RDMA device and jump to step S303.

[0026] The present invention also proposes a system for sharing RDMA devices in a Linux namespace, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for sharing RDMA devices in the Linux namespace.

[0027] The present invention also proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the method for sharing an RDMA device in a Linux namespace through a processor.

[0028] The present invention also proposes a computer program product, comprising a computer program / instruction, wherein the computer program / instruction is programmed or configured to execute the method for sharing an RDMA device in a Linux namespace through a processor.

[0029] Compared with the prior art, the advantages of the present invention are:

[0030] The present invention modifies the Linux namespace record data structure of the RDMA device into a list, so that it supports one RDMA device to be associated with multiple specified Linux namespaces at the same time. For one Linux namespace, all RDMA devices associated with this Linux namespace can be used. For one RDMA device, it can be associated with multiple Linux namespaces at the same time, thereby solving the problem that the original sharing mode cannot specify to use only specific RDMA devices, and solving the problem that the number of containers supported by the exclusive mode is limited. In addition, the present invention also adds a new partial sharing mode on the basis of the original sharing mode and exclusive mode, and modifies the operations of the RDMA device joining the Linux namespace and exiting the Linux namespace to make it compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode, without switching the mode in different scenarios, thereby solving the problem that the device may need to be restarted when changing the mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0032] Figure 2 This is a flow chart of adding an RDMA device to a namespace in an embodiment of the present invention.

[0033] Figure 3 This is a flowchart of an RDMA device exiting a namespace in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiment 1

[0035] This embodiment proposes a method for sharing RDMA devices in a Linux namespace, which can share RDMA devices in a specified Linux namespace list. For a Linux namespace, all RDMA devices associated with this Linux namespace can be used. For an RDMA device, it can be associated with multiple Linux namespaces at the same time, solving the problem that the original sharing mode cannot specify the use of only specific RDMA devices, and solving the problem that the number of containers supported by the exclusive mode is limited. Figure 1 As shown, the following steps are included:

[0036] For the RDMA mode of sharing RDMA devices between Linux namespaces, a new partial sharing mode (partshared) is added on the basis of the original sharing mode and exclusive mode, and the Linux namespace record data structure of the RDMA device is modified to support an RDMA device to be associated with multiple specified Linux namespaces at the same time, and the operations of RDMA devices joining and exiting the Linux namespace are modified to make them compatible with the sharing mode, exclusive mode and the newly added partial sharing mode.

[0037] In this embodiment, modifying the Linux namespace record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux namespaces at the same time means modifying the Linux namespace record data structure of the RDMA device into a list, and listing the specified Linux namespace information to which the RDMA device is simultaneously associated through the list, so that the RDMA device supports being associated with multiple Linux namespaces at the same time, and each Linux namespace can only discover and use the RDMA device recorded in the list according to the list.

[0038] Through the above steps, this embodiment adds a third mode, the partial sharing mode partshared, on the basis of the original shared mode shared and exclusive mode exclusive. The new partial sharing mode supports sharing RDMA devices in a list of specified Linux namespaces, that is, supports one RDMA device to be associated with multiple Linux namespaces at the same time.

[0039] For the original shared mode shared, since all RDMA devices are shared among all Linux namespaces, there is no need for the device to join and exit the Linux namespace namespace process. For the original exclusive mode exclusive, joining a Linux namespace namespace will automatically exit the original Linux namespace namespace, and only the device needs to join the Linux namespace namespace process. Therefore, in this embodiment, when modifying the operations of RDMA devices joining the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, exclusive mode and the newly added partial sharing mode, the main thing is to modify the original RDMA device joining the Linux namespace namespace process, increase support for the partial sharing mode partshared, and add the RDMA device exiting the Linux namespace namespace process.

[0040] Specifically, in this embodiment, after modifying the operations of the RDMA device joining the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device joining the Linux namespace are as follows: Figure 2 As shown, including:

[0041] S101, generating a network link request for modifying the Linux namespace of the RDMA device. In this embodiment, the generated network link request includes a setting command CMD_SET of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux namespace;

[0042] S102, sending a network connection request to the Linux kernel driver;

[0043] S103, the Linux kernel driver determines the RDMA mode of the shared RDMA device between the current Linux namespaces. If the RDMA mode is a shared mode, an error code -EOPNOTSUPP is returned, and the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partially shared mode, the process jumps to step S105;

[0044] S104, stop the RDMA device, modify the Linux namespace record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the Linux namespace originally joined, and then ends and exits;

[0045] S105, modify the Linux namespace record data structure of the RDMA device, keep the original Linux namespace unchanged, add a new Linux namespace, end and exit.

[0046] In this embodiment, after modifying the operations of the RDMA device joining the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device exiting the Linux namespace are as follows: Figure 3 As shown, including:

[0047] S201, generating a network link request to delete the RDMA device from a Linux namespace. In this embodiment, the generated network link request includes a deletion command CMD_DEL of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux namespace;

[0048] S202, sending a network connection request to the Linux kernel driver;

[0049] S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux namespaces. If it is a shared mode or an exclusive mode, an error code is returned, and the process ends and exits. If it is a partial shared mode, the process jumps to step S204;

[0050] S204, determine whether the requested Linux namespace exists in the Linux namespace record data structure of the RDMA device. If it does not exist in the Linux namespace record data structure of the RDMA device, that is, it does not exist in the list, return an error code, end and exit, otherwise, delete the requested Linux namespace from the Linux namespace record data structure of the RDMA device, that is, if it exists in the list, delete the requested Linux namespace from the list, end and exit.

[0051] Embodiment 2

[0052] This embodiment provides a method for sharing an RDMA device in a Linux namespace, which specifically includes the following steps:

[0053] S301, pre-allocate and configure the corresponding RDMA device for each Linux namespace, and record it in the initial value of the Linux namespace record data structure of each RDMA device;

[0054] S302, after each Linux namespace is started, discovering and using the RDMA devices available for the Linux namespace according to the content of the Linux namespace record data structure of each RDMA device, so that each RDMA device is only used by the Linux namespace recorded in its Linux namespace record data structure;

[0055] S303, during the operation of each Linux namespace, if it is necessary to modify the operation of the RDMA device joining the Linux namespace or exiting the Linux namespace, the operation of modifying the RDMA device joining the Linux namespace or exiting the Linux namespace in Example 1 is executed to update the Linux namespace record data structure of the RDMA device and jump to step S303.

[0056] Embodiment 3

[0057] This embodiment provides a system for sharing RDMA devices in a Linux namespace, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace described in the first embodiment.

[0058] This embodiment further proposes a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace described in Embodiment 1 through a processor.

[0059] This embodiment also proposes a computer program product, including a computer program / instruction, which is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace described in the first embodiment through a processor.

[0060] In summary, the present invention discloses a method and system for sharing RDMA devices in a Linux namespace. By adding a partial sharing mode and modifying the Linux namespace record data structure of the RDMA device to a list, the problem of being unable to specify to use only a specific RDMA device when using the sharing mode is solved, and the problem of the limited number of containers supported in the exclusive mode is solved. Compared with the prior art, the advantages of the present invention are:

[0061] (1) It not only supports sharing of RDMA devices among multiple Linux namespaces, retaining the characteristics of the sharing mode, but also supports controlling the list of shared Linux namespaces. In the scenario with multiple RDMA devices, it can support the Linux namespace to only discover and use specific RDMA devices to achieve optimal performance.

[0062] (2) It supports controlling the sharing of a single RDMA device in a Linux namespace list. Compared with the exclusive mode, which only supports the use of one RDMA device in one Linux namespace, the number of containers supported by the server depends on the number of RDMA devices and the number of virtual functions that can be virtualized by a single RDMA. It can support the running of more containers, improve the deployment density of containers and the utilization rate of servers. Moreover, compared with the exclusive mode, in which some single-root input and output virtualization sriov configuration modifications of the RDMA device need to be written into the device firmware and the server needs to be restarted, the present invention does not need to use sriov and does not have this problem.

[0063] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for sharing RDMA devices in a Linux namespace, characterized in that: The invention comprises the following steps: adding a new partial sharing mode to an RDMA mode for sharing RDMA devices between Linux namespaces on the basis of the original sharing mode and exclusive mode, modifying the Linux namespace record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple specified Linux namespaces at the same time, and modifying the operations of the RDMA device to join the Linux namespace and exit the Linux namespace so that it is compatible with the sharing mode, the exclusive mode and the newly added partial sharing mode; modifying the Linux namespace record data structure of the RDMA device so that it supports one RDMA device to be associated with multiple Linux namespaces at the same time means modifying the Linux namespace record data structure of the RDMA device into a list, and listing the specified Linux namespace information to which the RDMA device is associated at the same time through the list, so that the RDMA device supports being associated with multiple Linux namespaces at the same time, and each Linux namespace can only discover and use the RDMA device recorded in the list according to the list.

2. The method for sharing RDMA devices in a Linux namespace according to claim 1, characterized in that: After the operations of modifying the RDMA device to join the Linux namespace and exit the Linux namespace to make it compatible with the shared mode, the exclusive mode and the newly added partial shared mode, the operations of joining the RDMA device to the Linux namespace include: S101, generating a network link request for modifying the Linux namespace of the RDMA device; S102, sending a network connection request to the Linux kernel driver; S103, the Linux kernel driver determines the RDMA mode of the shared RDMA device between the current Linux namespaces. If the RDMA mode is a shared mode, an error code is returned, and the process ends and exits. If it is an exclusive mode, the process jumps to step S104; if it is a partial shared mode, the process jumps to step S105; S104, stop the RDMA device, modify the Linux namespace record data structure of the RDMA device, rename the device name of the RDMA device, and recreate the attribute file of the RDMA device, so that the RDMA device enters the exclusive mode and automatically exits the Linux namespace originally joined, and then ends and exits; S105, modify the Linux namespace record data structure of the RDMA device, keep the original Linux namespace unchanged, add a new Linux namespace, end and exit.

3. The method for sharing RDMA devices in a Linux namespace according to claim 2, characterized in that: The network link request generated in step S101 includes a setting command CMD_SET of the network link request, an index INDEX of the RDMA device corresponding to the network link request, and a file descriptor fd of the Linux namespace.

4. The method for sharing RDMA devices in a Linux namespace according to claim 2, wherein: After modifying the operations of adding the RDMA device to the Linux namespace and exiting the Linux namespace to make it compatible with the shared mode, the exclusive mode, and the newly added partial shared mode, the operations of the RDMA device exiting the Linux namespace include: S201, generating a network link request to delete the RDMA device from a Linux namespace; S202, sending a network connection request to the Linux kernel driver; S203, the Linux kernel driver determines the RDMA mode of the RDMA device shared between the current Linux namespaces. If it is a shared mode or an exclusive mode, an error code is returned, and the process ends and exits. If it is a partial shared mode, the process jumps to step S204; S204, determine whether the requested Linux namespace exists in the Linux namespace record data structure of the RDMA device. If it does not exist in the Linux namespace record data structure of the RDMA device, return an error code, end and exit, otherwise, delete the requested Linux namespace from the Linux namespace record data structure of the RDMA device, end and exit.

5. The method for sharing RDMA devices in a Linux namespace according to claim 4, characterized in that: The network connection request generated in step S201 includes a delete command CMD_DEL of the network connection request, an index INDEX of the RDMA device corresponding to the network connection request, and a file descriptor fd of the Linux namespace.

6. The method for sharing RDMA devices in a Linux namespace according to claim 5, characterized in that: include: S301, pre-allocate and configure the corresponding RDMA device for each Linux namespace, and record it in the initial value of the Linux namespace record data structure of each RDMA device; S302, after each Linux namespace is started, discovering and using the RDMA devices available for the Linux namespace according to the content of the Linux namespace record data structure of each RDMA device, so that each RDMA device is only used by the Linux namespace recorded in its Linux namespace record data structure; S303, during the operation of each Linux namespace, if it is necessary to modify the operation of the RDMA device joining the Linux namespace or exiting the Linux namespace, the operation of modifying the RDMA device joining the Linux namespace or exiting the Linux namespace is executed to update the Linux namespace record data structure of the RDMA device and jump to step S303.

7. A system for sharing RDMA devices in a Linux namespace, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace as described in any one of claims 1 to 6 through a processor.

9. A computer program product comprising a computer program / instructions, characterized in that The computer program / instruction is programmed or configured to execute the method for sharing RDMA devices in a Linux namespace as described in any one of claims 1 to 6 through a processor.

Images