A container storage method, system, device, medium, and product

Abstract

The application discloses a container storage method, system, device, medium and product, and selects a first node according to Pod storage requirements and storage capacities of nodes; in the first node, a target end is created through a target end service NVMe-oF Target based on an Ethernet nonvolatile fast memory storage protocol; a second node deployed by the Pod is acquired, in the second node, an initiator end connected with the target end is created through an initiator end service NVMe-oF Initiator based on the Ethernet nonvolatile fast memory storage protocol; a local storage interface is generated in the second node through the initiator end; the local storage interface is mapped to storage resources of the target end; and the local storage interface is mounted on a file directory of the Pod. According to the embodiment of the application, PCIe local channels and NVMe-oF remote channels can be simultaneously supported to adapt to a multi-container scene.

Description

Technical Field

[0001] This invention relates to the field of container storage technology, and in particular to a container storage method, system, device, medium and product. Background Technology

[0002] Kubernetes, as an open-source container orchestration platform, enables efficient management of large-scale container clusters. Furthermore, Kubernetes boasts strong extensibility, allowing developers to customize container storage plugins through standardized interfaces, bridging the gap between native storage functionality and actual business needs.

[0003] However, current container storage solutions have significant shortcomings in terms of scenario adaptability: for example, solutions based on file systems such as Ceph are limited by the characteristics of the file system architecture and are difficult to adapt to the high-performance storage requirements of container scenarios; another example is the node affinity solution, which significantly increases the difficulty of service recovery when a node fails, thereby affecting the overall availability of the service. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a container storage method, system, device, medium, and product that can simultaneously support PCIe local access and NVMe-oF remote access to adapt to multi-container scenarios.

[0005] In a first aspect, embodiments of the present invention provide a container storage method, comprising:

[0006] Based on the Pod's storage requirements and the storage capacity of each node, select the first node;

[0007] In the first node, a target is created using the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol;

[0008] Obtain the second node deployed by the Pod. On the second node, create an initiator that connects to the target node through the NVMe-oF Initiator service based on the Ethernet-based non-volatile fast storage protocol.

[0009] A local storage interface is generated on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end;

[0010] Mount the local storage interface to the Pod's file directory.

[0011] As an improvement to the above solution, the step of selecting the first node based on the Pod's storage requirements and the storage capacity of each node includes:

[0012] The storage capacity of each node is obtained by deploying the CSI node plugin on each node;

[0013] The storage capacity of each node is obtained by using the CSI control plugin deployed on the Kubernetes cluster, based on the storage capacity.

[0014] The Kubernetes scheduler selects the first node from all nodes based on the available storage and the Pod's storage requirements.

[0015] As an improvement to the above scheme, the step of creating a target terminal in the first node through the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol includes:

[0016] Obtain storage volume request information, and the first node creates a logical volume that matches the storage volume request information through Bdev;

[0017] Bind the logical volume to NVMe-oF transport layer parameters;

[0018] Based on the bound logical volume, initialize the NVMe-oF Target service, expose the storage service of the logical volume, and generate a target end that can be identified by the initiating end; the storage resource of the target end is the logical volume.

[0019] As an improvement to the above solution, the step of obtaining storage volume request information, in which the first node creates a logical volume matching the storage volume request information via Bdev, includes:

[0020] Obtain storage volume request information from the CSI controller plugin;

[0021] Using the CSI node plugin of the first node, a logical volume matching the storage volume request information is created in the storage capacity of the first node via Bdev.

[0022] As an improvement to the above solution, a second node deployed by the Pod is obtained. On this second node, an initiator service (NVMe-oF Initiator) based on the Ethernet-based non-volatile fast storage protocol is created to connect with the target node, including:

[0023] Based on the first node, the Pod is scheduled to obtain the second node where the Pod is deployed;

[0024] Based on the obtained storage resource information of the target end, configure the NVMe-oF Initiator service parameters and create the initiator through the SPDK API;

[0025] A connection request is initiated to the target terminal, thereby establishing a storage access channel between the target terminal and the initiating terminal.

[0026] As an improvement to the above solution, the step of scheduling the Pod based on the first node to obtain the second node where the Pod is deployed includes:

[0027] Obtain the Pod's runtime requirements, and use the Kubernetes scheduler to select a third node from all nodes that meets the Pod's runtime requirements;

[0028] If the third node includes the first node, increase the scoring weight of the first node;

[0029] Calculate the operational quality score of the third node based on the scoring weights.

[0030] Based on the operational quality score of the third node, the second node is selected for Pod scheduling.

[0031] As an improvement to the above scheme, if the third node includes the first node, increasing the scoring weight of the first node includes:

[0032] If the third node includes the first node, the scheduler plugin is invoked via gRPC to increase the scoring weight of the first node; the scheduler plugin is deployed in the extension layer of the Kubernetes scheduler.

[0033] As an improvement to the above solution, the step of generating a local storage interface on the second node through the initiating end includes:

[0034] Using the CSI node plugin of the second node, a virtual block device is created in the storage capacity of the second node through Bdev, which maps to the storage resources of the target end;

[0035] A local storage interface is generated based on the virtual block device.

[0036] As an improvement to the above solution, mounting the local storage interface to the Pod's file directory includes:

[0037] Generate a unique file directory for the Pod on the second node using the Kubernetes Kubelet component;

[0038] The local storage interface is mounted to the unique file directory via the CSI node plugin.

[0039] As an improvement to the above solution, if the Pod's file directory is already mounted, then when the Pod generates application data, the first node and the second node are obtained.

[0040] If the first node and the second node are the same, the application data can be directly read and written to the target storage resources through SPDK;

[0041] If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target end for reading and writing via the TCP protocol.

[0042] Secondly, embodiments of the present invention also provide a container storage system, comprising:

[0043] The node filtering module is used to select the first node based on the Pod's storage requirements and the storage capacity of each node;

[0044] The target creation module is used to create a target in the first node through the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol;

[0045] The initiator creation module is used to obtain the second node where the Pod is deployed. On the second node, an initiator is created to connect with the target node through the NVMe-oF Initiator service based on the Ethernet non-volatile fast storage protocol.

[0046] A container binding module is used to generate a local storage interface on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end;

[0047] The container mounting module is used to mount the local storage interface to the file directory of the Pod.

[0048] Thirdly, embodiments of the present invention provide a container storage device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the container storage method described above.

[0049] Fourthly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the container storage method as described above.

[0050] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program or computer instructions, wherein when the computer program or computer instructions are executed by a processor, the container storage method described above is performed.

[0051] Compared with existing technologies, the container storage method, system, device, medium, and product disclosed in this invention select a first node based on the Pod's storage requirements and the storage capacity of each node. On the first node, a target endpoint is created using the NVMe-oF Target service based on the Ethernet-based Non-Volatile Fast Memory Storage Protocol. A second node deployed by the Pod is obtained. On the second node, an initiator service based on the Ethernet-based Non-Volatile Fast Memory Storage Protocol, NVMe-oF Initiator, is created to connect to the target endpoint. A local storage interface is generated on the second node through the initiator. The local storage interface is mapped to the storage resources of the target endpoint. The local storage interface is mounted on the Pod's file directory. Using this embodiment, both PCIe local paths and NVMe-oF remote paths can be supported simultaneously to adapt to multi-container scenarios. Attached Figure Description

[0052] Figure 1 This is a schematic flowchart of a container storage method provided in an embodiment of the present invention;

[0053] Figure 2 This is an architecture diagram of the capacity storage plug-in provided in an embodiment of the present invention;

[0054] Figure 3 This is a scheduling flowchart at the container storage plugin level provided in an embodiment of the present invention;

[0055] Figure 4 This is a flowchart illustrating the steps of using a mounted data volume in a Pod, as provided in an embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram of the structure of a container storage system provided in an embodiment of the present invention;

[0057] Figure 6 This is a structural block diagram of a container storage device provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] In the description and claims, it should be understood that the terms "first," "second," etc., used in the description and claims are only for the purpose of distinguishing the description of the same technical features, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated, nor necessarily the order of description or chronological order. The terms are interchangeable where appropriate. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0060] The native Kubernetes platform manages and controls multiple nodes in the cluster server through components such as API Server (Application Programming Interface Service), Scheduler, Controller Manager, and Kubelet (node ​​agent). API Server is responsible for receiving and processing all resource operation requests, such as Pod creation requests and storage creation requests; Scheduler is responsible for binding unscheduled Pods to appropriate nodes; Controller Manager manages various controllers; and Kubelet is deployed on each node, taking over the full lifecycle management of Pods on that node.

[0061] Kubernetes native components provide basic management capabilities for cluster servers, but there are still gaps between them and actual business needs, requiring the development of container storage plugins to bridge the gap. Current container storage plugins mainly include adaptive plugins and cloud-native plugins.

[0062] Adaptive plugins adapt to distributed file systems by integrating the Container Storage Interface (CSI). Examples include ceph-csi for Ceph and gluster-heketi-csi-driver for GlusterFS. Cloud-native plugins, on the other hand, are designed for container scenarios, building dedicated storage architectures around container requirements. For instance, Longhorn, based on the iSCSI protocol, manages storage volumes on the control plane using a self-developed system combined with container specifications, and enables cross-node data writing on the data plane via the iSCSI protocol. Another example is OpenEBS, which adds node affinity to data volumes through LVM, hostpath, and other methods, thereby providing efficient and simple local storage.

[0063] The performance of existing adaptive plugins is limited by the file system itself. For example, Longhorn's cloud-native plugin is also limited by the performance of iSCSI technology and cannot meet the performance requirements of high-performance storage in container scenarios. Although cloud-native plugins based on hostpath can meet the high-performance requirements, their nodes are bound to affinity, which will significantly increase the difficulty of service recovery when nodes fail, thus affecting the overall availability of the service.

[0064] Current container storage solutions all have their own problems to varying degrees, and none of them can provide a complete high-performance storage solution suitable for multi-container scenarios.

[0065] Based on the above considerations, embodiments of the present invention provide a container storage method. Please refer to... Figure 1 In this embodiment, the container storage method is specifically executed through steps S1 to S5:

[0066] S1. Select the first node based on the Pod's storage requirements and the storage capacity of each node.

[0067] S2. In the first node, a target is created using the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol.

[0068] S3. Obtain the second node deployed by the Pod. On the second node, create an initiator that connects to the target node through the NVMe-oF Initiator service, which is based on the Ethernet-based non-volatile fast storage protocol.

[0069] S4. A local storage interface is generated on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end.

[0070] S5. Mount the local storage interface to the Pod's file directory.

[0071] It should be noted that mounting a Pod's file directory means associating the Pod with a certain type of storage resource, enabling the Pod to persistently read and write data through that directory.

[0072] NVMe-oF (Non-Volatile Memory Express over Fabrics) is a remote storage technology that extends the NVM storage protocol. It extends local NVMe to remote storage devices through the Fabric network, allowing remote nodes to access remote storage resources with low latency and high bandwidth, just like accessing a local PCIe SSD.

[0073] The core function of the NVMe-oF Target service, a non-volatile fast storage protocol based on Ethernet, is to encapsulate the physical storage on a node into a target that can be shared via the NVMe-oF network and to receive access requests from other nodes. The core function of the NVMe-oF Initiator service, also based on the NVMe-oF protocol, is to proactively initiate connection requests to the NVMe-oF Target, establish a remote storage path, and recognize the remote storage provided by the target as a locally accessible storage device.

[0074] Preferably, NVMe-oF is built on SPDK (Storage Performance Development Kit).

[0075] It should be noted that the first node is the node that provides storage resources for the Pod, and the second node is the node where the Pod is actually deployed and runs, and is also the node where the storage resource access end is located. The storage needs of the Pod are met through the interaction between the first node and the second node.

[0076] In some preferred embodiments, the first node and the second node are the same node in the cluster server, and the interaction between the first node and the second node is achieved through the PCIe local path; in other preferred embodiments, the first node and the second node are different nodes in the cluster server, and the interaction between the nodes is achieved through the NVMe-oF remote path, with the physical storage volume of the Pod resource located on the first node.

[0077] In this embodiment of the invention, NVMe over Fabrics technology is used to provide the ability for cross-node storage sharing of containers. On this basis, it can simultaneously support PCIe local access and NVMe-oF remote access, so that the data disk on the cluster server can be accessed directly by local applications and can also be accessed by other nodes through the network to adapt to multi-container scenarios.

[0078] Preferably, NVMe-oF connects to Kubernetes via the CSI (Container Storage Interface) plugin to provide high-speed persistent storage for Kubernetes containers.

[0079] It should be noted that CSI is a currently universally accepted standard specification in the industry. Its goal is to unify the interface between container orchestration platforms and underlying storage systems. It defines a set of common APIs for operations such as creating and mounting volumes. By developing plugins that conform to the CSI standard, it is possible to adapt to all container platforms that support CSI.

[0080] Preferably, in this embodiment of the invention, a CSI control plugin and a CSI node plugin are constructed.

[0081] The CSI control plugin implements the Container Storage Plugin Specification's Controller section, responsible for the pre-container aspects of container creation, including metadata management and invoking application node plugins. It's important to note that only one replica in the cluster provides service at any given time.

[0082] The CSI node plugin is deployed on each node, implementing the Node part of the Container Storage Plugin Specification. It is responsible for the actual creation of storage and binding with Pods, and provides an interface for CSI control plugins to call.

[0083] Please see Figure 2 The Container Storage Interface (CSI) control plugin can call the Container Storage Interface (CSI) node plugin of the application node to create a High Performance Storage Development Kit (SPDK) target or initiator. The CSI node plugin can create a virtual block through the SPDK's Block Device Layer (Bdev) and then create the target based on the SPDK's NVMe-oF Target service, or create the initiator based on the SPDK's NVMe-oF Initiator service.

[0084] As a preferred implementation, step S1, selecting the first node based on the Pod's storage requirements and the storage capacity of each node, includes:

[0085] The storage capacity of each node is obtained by deploying the CSI node plugin on each node;

[0086] The storage capacity of each node is obtained by using the CSI control plugin deployed on the Kubernetes cluster, based on the storage capacity.

[0087] The Kubernetes scheduler selects the first node from all nodes based on the available storage and the Pod's storage requirements.

[0088] In practical applications, the CSI node plugin on each node continuously collects and reports its own storage capacity. Running locally on the node, the CSI node plugin can detect changes in storage capacity with low latency. The CSI control plugin aggregates the storage capacity information of all nodes, determines which nodes have available storage space, and then feeds this information back to Kubernetes. During the volume scheduling phase, the Kubernetes scheduler selects the first node from the candidate list provided by the CSI control plugin, taking into account multi-dimensional constraints.

[0089] As a preferred implementation, step S2, in the first node, creates a target endpoint through the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol, including:

[0090] Obtain storage volume request information, and the first node creates a logical volume that matches the storage volume request information through Bdev;

[0091] Bind the logical volume to NVMe-oF transport layer parameters;

[0092] Based on the bound logical volume, initialize the NVMe-oF Target service, expose the storage service of the logical volume, and generate a target end that can be identified by the initiating end; the storage resource of the target end is the logical volume.

[0093] Preferably, the storage volume request information includes first node information, storage volume type information, and / or storage volume size information.

[0094] Further, preferably, the step of obtaining storage volume request information, in which the first node creates a logical volume matching the storage volume request information via Bdev, includes:

[0095] Obtain storage volume request information from the CSI controller plugin;

[0096] Using the CSI node plugin of the first node, a logical volume matching the storage volume request information is created in the storage capacity of the first node via Bdev.

[0097] In this embodiment of the invention, after selecting the first node, the CSI control plugin distributes the storage volume request information to the CSI node plugin of the first node. It should be noted that the CSI control plugin does not directly operate the node hardware.

[0098] Next, the CSI node plugin of the first node efficiently creates a logical volume matching the storage volume request information through Bdev (Block Device), a core component of the SPDK framework. In this embodiment of the invention, the logical volume is an independent block device partition on a physical NVMe SSD. SPDK's Bdev reduces the overhead of switching between kernel mode and user mode, improving IO performance.

[0099] The Transport parameter is the NVMe-oF communication address, defining the network path that the Target service provides for external access. By binding the logical volume to the Transport parameter, reachability for cross-boundary access is ensured.

[0100] Initializing the NVMe-oF Target service involves encapsulating a logical volume with bound transport layer parameters into a remote block device service conforming to the NVMe-oF protocol, generating the target endpoint, and ensuring that subsequent initiating endpoints can accurately identify and connect to this target endpoint.

[0101] As a preferred implementation, step S3 involves obtaining the second node deployed by the Pod. On this second node, an initiator service (NVMe-oF Initiator) is created to connect with the target node, including:

[0102] Based on the first node, the Pod is scheduled to obtain the second node where the Pod is deployed;

[0103] Based on the obtained storage resource information of the target end, configure the NVMe-oF Initiator service parameters and create the initiator through the SPDK API;

[0104] A connection request is initiated to the target terminal, thereby establishing a storage access channel between the target terminal and the initiating terminal.

[0105] After creating the volume using the API provided by the CSI specification in step S2, the Pod needs to be scheduled, entering the Pod scheduling phase. In the traditional scheduling process, the Kubernetes scheduler considers the running quality of each node and selects a second node for the Pod to be deployed and run on.

[0106] It should also be noted that, in this preferred embodiment, during the Pod scheduling phase, the selected first node is considered. For example, a special identifier is provided for the first node so that the Kubernetes scheduler will give priority to the first node or a node in the same network segment and rack as the first node as the second node, thereby reducing the network latency of the NVMe-oF remote path.

[0107] The CSI control plugin sends information about the target's storage resources to the CSI node plugin on the second node, enabling the CSI node plugin to create an initiator based on this information using the SPDK-based NVMe-oF Initiator service.

[0108] By sending an NVMe-oF connection request from the initiator to the target, bidirectional authentication and path establishment between the first node (Target) and the second node (Initiator) can be completed, enabling the second node to have a physical channel to access the storage resources of the first node.

[0109] Further, preferably, the step of scheduling the Pod based on the first node to obtain the second node where the Pod is deployed includes:

[0110] Obtain the Pod's runtime requirements, and use the Kubernetes scheduler to select a third node from all nodes that meets the Pod's runtime requirements;

[0111] If the third node includes the first node, increase the scoring weight of the first node;

[0112] Calculate the operational quality score of the third node based on the scoring weights.

[0113] Based on the operational quality score of the third node, the second node is selected for Pod scheduling.

[0114] Based on NVMe-oF technology, the Pod running node and storage node can be separated. In a preferred embodiment of the present invention, considering that data transmission on the same node has higher real-time performance than transmission between different nodes, the selected first node is further considered during the Pod scheduling phase to ensure that the Pod's running service and storage service are located on the same node as much as possible.

[0115] In some preferred embodiments, the present invention also constructs a scheduler plugin called via gRPC. This scheduler plugin is an extension of the Kubernetes scheduler and is responsible for prioritizing the selection of nodes containing storage during Kubernetes scheduling. Specifically, it extends the score process by favoring nodes containing storage, thereby influencing the node selection priority.

[0116] By extending the scheduler plugin, the Pod scheduling phase uses a local-first scheduling algorithm to keep the storage volume and Pod node on the same node as much as possible. This allows data to be written directly to the local data disk without crossing the network, thus shortening the data read and write path.

[0117] Furthermore, preferably, if the third node includes the first node, increasing the scoring weight of the first node includes:

[0118] If the third node includes the first node, the scheduler plugin is invoked via gRPC to increase the scoring weight of the first node; the scheduler plugin is deployed in the extension layer of the Kubernetes scheduler.

[0119] In some preferred embodiments, the scheduler plugin operates by assigning weights to all nodes on the volume containing the first node based on the information of the first node provided by the CSI control plugin when the Kubernetes scheduler performs scoring. The weight of each node is determined based on its proximity to the first node.

[0120] As a preferred implementation, step S4, generating a local storage interface on the second node through the initiating end, includes:

[0121] Using the CSI node plugin of the second node, a virtual block device is created in the storage capacity of the second node through Bdev, which maps to the storage resources of the target end;

[0122] A local storage interface is generated based on the virtual block device.

[0123] It should be noted that the virtual block device created by the second node through Bdev is a logical mapping of the storage resources of the first node. It occupies the device node identifier of the second node but does not consume the local persistent storage of the second node. Through the mapping relationship between the virtual block device and the storage resources, the storage data is persisted to disk in the storage resources of the first node.

[0124] By generating a local storage interface, Pods can recognize and mount objects inside SPDK. Pods do not need to know whether the storage is a local PCIe disk or a remote NVMe-oF volume; they can simply access it through the standard path.

[0125] As a preferred implementation, step S5, mounting the local storage interface to the Pod's file directory, includes:

[0126] Generate a unique file directory for the Pod on the second node using the Kubernetes Kubelet component;

[0127] The local storage interface is mounted to the unique file directory via the CSI node plugin.

[0128] The second node may run multiple Pods simultaneously. The Kubelet component creates a unique file directory for each Pod and sets access permissions for it, which can coordinate the storage data of each Pod. In addition, as a native component responsible for Pod lifecycle management, the Kubelet component can also coordinate the interaction timing of components such as CSI plugins to ensure that the container storage process proceeds in an orderly manner.

[0129] In the above solution, by mounting the local storage interface to a unique file directory, the application in the Pod faces a standard file system directory. It does not need to be aware of the details such as the actual data being stored on the remote node or the underlying storage application using the NVMe-oF protocol based on SPDK. It can achieve the requirements of writing data to remote storage and reading remote data just like reading and writing a local directory, thus reducing the difficulty of developing and maintaining Pod applications.

[0130] In some preferred embodiments, when a Pod is deleted, the Kubelet component triggers the corresponding CSI plugin to perform an unloading operation and clean up the unique file directory. This embodiment of the invention can automatically mount and unmount Pods as they start and stop, ensuring that storage resources are synchronized with the Pod lifecycle and improving the utilization of cluster resources.

[0131] Please see Figure 3 , Figure 3 A scheduling flowchart at the container storage plugin level is provided based on the CSI specification.

[0132] When the Application Programming Interface (API) Server receives a PVC (Persistent Volume Claim) and a Pod submitted by a user, it enters the volume scheduling phase. The Kubernetes native Controller Manager listens to the PVC and determines whether the PVC has a corresponding PV (Persistent Volume).

[0133] If no corresponding PV exists, the volume creation phase begins. The External Provisioner listens for PVC events and sends volume creation commands to the CSI control plugin created in this embodiment. The CSI control plugin coordinates the CSI node plugin of the first node to create the SPDK Target. The CSI node plugin abstracts the physical NVMe disk into a block device manageable by the SPDK through virtual blocks (Bdev), configures the NVMe TCP Transport layer, enables NVMe-oF network transmission capabilities, and then registers the target SPDK Target to provide NVMe-oF storage services. Once the volume is successfully created, the External Provisioner notifies the API Server to create the PV to complete the binding between the PVC and the PV.

[0134] After the volume creation phase is completed, the Pod scheduling phase begins. The Kubernetes scheduler reads the SPDK storage volume information to score all candidate nodes. During the scoring process, the scheduler plugin created in this embodiment of the invention is called to adjust the weight score of the first node. Finally, the scored nodes are returned to the Kubernetes scheduler to determine the second node where the Pod resides.

[0135] Once the node hosting the Pod is determined, the volume binding phase can begin. The CSI node plugin on the second node creates a client-side SPDK Initiator, then establishes a remote storage path via the NVMe-oF protocol, connecting to the target SPDK on the first node. A virtual block device is then created locally and mapped to the corresponding remote physical volume storage. At this point, the second node can access the storage resources of the first node through the client-side SPDK Initiator.

[0136] Once the virtual block device and the Pod are ready, the CSI node plugin can bind them together. During the volume mounting phase, Kubernetes prepares a unique file directory for the Pod to mount onto the virtual block device, after which the Pod can correctly use the corresponding block storage.

[0137] The above steps S1-S5 complete the entire PVC and Pod creation process. After that, the Pod can start normally and use the mounted data volume. Subsequent data reading and writing for the file system will be handled through SPDK.

[0138] As a preferred implementation, if the Pod's file directory has been mounted using the above embodiments, the container storage method is specifically executed through steps S6 to S8:

[0139] S6. When a Pod generates application data, retrieve the first and second nodes;

[0140] S7. If the first node and the second node are the same, the application data is directly read and written to the target end's storage resources through SPDK.

[0141] S8. If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target end for reading and writing via the TCP protocol.

[0142] Preferably, the NVMe commands include NVMe storage commands and NVMe read commands.

[0143] Taking data writing as an example, there are two scenarios based on the consistency between the Pod and the storage node.

[0144] In most cases, the Pod and storage are on the same node. In this case, data can be written directly to the created logical data disk through SPDK without going through the TCP / IP protocol stack, resulting in no network loss.

[0145] In some cases, the Pod and storage are not on the same node, for example, due to high CPU load on a node, causing the application to be moved to another node. In such cases, data transfer across the network is required. Please see [link to relevant documentation]. Figure 4 After the application data of the Pod is written to the virtual block device through the file system, the data in the virtual block storage is converted into NVMe storage commands, and then sent to the remote NVMe target through the TCP protocol. After receiving the storage commands, the target writes them into the associated physical storage medium.

[0146] The container storage method provided by this invention adopts NVMe over Fabrics technology, which provides the ability for cross-node storage sharing of containers and can adapt to multi-container scenarios. On this basis, it can simultaneously support PCIe local access and NVMe-oF remote access, so that the data disk on the cluster server can be accessed directly by local applications and can also be accessed by other nodes through the network. Through the design of computing and storage separation, it ensures rapid self-healing after node failures and other problems occur.

[0147] This invention provides a container storage system. Please refer to [link / reference]. Figure 5 The container storage system controls and schedules the node filtering module 11, target creation module 12, initiator creation module 13, container binding module 14, and container mounting module 15, wherein:

[0148] The node filtering module 11 is used to select the first node based on the Pod's storage requirements and the storage capacity of each node;

[0149] The target creation module 12 is used to create a target in the first node by means of the target service NVMe-oF Target based on the Ethernet-based non-volatile fast memory storage protocol.

[0150] The initiator creation module 13 is used to obtain the second node deployed by the Pod. In the second node, an initiator is created to connect with the target node through the initiator service NVMe-oF Initiator based on Ethernet non-volatile fast storage protocol.

[0151] The container binding module 14 is used to generate a local storage interface on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end;

[0152] The container mounting module 15 is used to mount the local storage interface to the file directory of the Pod.

[0153] In a preferred embodiment, the node filtering module 11 is specifically used for:

[0154] The storage capacity of each node is obtained by deploying the CSI node plugin on each node;

[0155] The storage capacity of each node is obtained by using the CSI control plugin deployed on the Kubernetes cluster, based on the storage capacity.

[0156] The Kubernetes scheduler selects the first node from all nodes based on the available storage and the Pod's storage requirements.

[0157] In a preferred embodiment, the target end creation module 12 includes:

[0158] A logical volume creation unit is used to obtain storage volume request information, and the first node creates a logical volume that matches the storage volume request information through Bdev;

[0159] A transport layer parameter binding unit is used to bind the logical volume to NVMe-oF transport layer parameters;

[0160] The target end generation unit is used to initialize the NVMe-oF Target service based on the bound logical volume, expose the storage service of the logical volume, and generate a target end that can be identified by the initiating end; the storage resource of the target end is the logical volume.

[0161] Further, preferably, the logical volume creation unit is specifically used for:

[0162] Obtain storage volume request information from the CSI controller plugin;

[0163] Using the CSI node plugin of the first node, a logical volume matching the storage volume request information is created in the storage capacity of the first node via Bdev.

[0164] In a preferred embodiment, the initiator creation module 13 includes:

[0165] The Pod scheduling unit is used to schedule Pods based on the first node to obtain the second node where the Pod is deployed;

[0166] The initiator creation unit is used to configure the NVMe-oFInitiator service parameters based on the obtained storage resource information of the target end, and create the initiator through the SPDK API;

[0167] A connection request unit is used to initiate a connection request to the target end, thereby forming a storage access channel between the target end and the initiating end.

[0168] Further, preferably, the Pod scheduling unit is specifically used for:

[0169] Obtain the Pod's runtime requirements, and use the Kubernetes scheduler to select a third node from all nodes that meets the Pod's runtime requirements;

[0170] If the third node includes the first node, increase the scoring weight of the first node;

[0171] Calculate the operational quality score of the third node based on the scoring weights.

[0172] Based on the operational quality score of the third node, the second node is selected for Pod scheduling.

[0173] Furthermore, preferably, if the third node includes the first node, increasing the scoring weight of the first node includes:

[0174] If the third node includes the first node, the scheduler plugin is invoked via gRPC to increase the scoring weight of the first node; the scheduler plugin is deployed in the extension layer of the Kubernetes scheduler.

[0175] In a preferred embodiment, the container binding module 14 is specifically used for:

[0176] Using the CSI node plugin of the second node, a virtual block device is created in the storage capacity of the second node through Bdev, which maps to the storage resources of the target end;

[0177] A local storage interface is generated based on the virtual block device.

[0178] In a preferred embodiment, the container mounting module 15 is specifically used for:

[0179] Generate a unique file directory for the Pod on the second node using the Kubernetes Kubelet component;

[0180] The local storage interface is mounted to the unique file directory via the CSI node plugin.

[0181] In a preferred embodiment, if the Pod's file directory is already mounted, the container storage system also controls and schedules the application data read / write module therein. The application data read / write module is used for:

[0182] When a Pod generates application data, retrieve the first and second nodes;

[0183] If the first node and the second node are the same, the application data can be directly read and written to the target storage resources through SPDK;

[0184] If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target end for reading and writing via the TCP protocol.

[0185] The container storage system provided by this invention adopts NVMe over Fabrics technology, which provides the ability for cross-node storage sharing of containers and can adapt to multi-container scenarios. On this basis, it can simultaneously support PCIe local access and NVMe-oF remote access, so that the data disk on the cluster server can be accessed directly by local applications and can also be accessed by other nodes through the network. Through the design of separating computing and storage, the file system can quickly self-heal after node failures and other problems occur.

[0186] See Figure 6 , Figure 6 This is a structural block diagram of a container storage device provided in an embodiment of the present invention. The container storage device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described container storage method embodiments, such as steps S1-S5 and / or S6-S8.

[0187] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the container storage device.

[0188] The container storage device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of a container storage device and does not constitute a limitation on the container storage device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the container storage device may also include input / output devices, network access devices, buses, etc.

[0189] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the container storage device, connecting various parts of the container storage device via various interfaces and lines.

[0190] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the container storage device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0191] Wherein, if the modules / units integrated in the container storage device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 31, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0192] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A container storage method, characterized in that, include: Based on the Pod's storage requirements and the storage capacity of each node, select the first node; The first node is the node that provides storage resources for the Pod; In the first node, a target is created using the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol; Obtain the second node deployed by the Pod. On the second node, create an initiator that connects to the target node through the NVMe-oF Initiator service based on the Ethernet-based non-volatile fast storage protocol. A local storage interface is generated on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end; Mount the local storage interface onto the Pod's file directory; If the Pod's file directory is already mounted, then when the Pod generates application data, obtain the first node and the second node; If the first node and the second node are the same, the application data can be directly read and written to the target storage resources through SPDK; If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target end for reading and writing via the TCP protocol; The step of obtaining the second node where the Pod is deployed includes: Obtain the Pod's runtime requirements, and use the Kubernetes scheduler to select a third node from all nodes that meets the Pod's runtime requirements; If the third node includes the first node, increase the scoring weight of the first node; Calculate the operational quality score of the third node based on the scoring weights. Based on the operational quality score of the third node, the second node is selected for Pod scheduling.

2. The container storage method as described in claim 1, characterized in that, The step of selecting the first node based on the Pod's storage requirements and the storage capacity of each node includes: The storage capacity of each node is obtained by deploying the CSI node plugin on each node; The storage capacity of each node is obtained by using the CSI control plugin deployed on the Kubernetes cluster, based on the storage capacity. The Kubernetes scheduler selects the first node from all nodes based on the available storage and the Pod's storage requirements.

3. The container storage method as described in claim 1, characterized in that, In the first node, creating a target endpoint through the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol includes: Obtain storage volume request information, and the first node creates a logical volume that matches the storage volume request information through Bdev; Bind the logical volume to NVMe-oF transport layer parameters; Based on the bound logical volume, initialize the NVMe-oF Target service, expose the storage service of the logical volume, and generate a target end that can be identified by the initiating end; the storage resource of the target end is the logical volume.

4. The container storage method as described in claim 3, characterized in that, The step of obtaining storage volume request information, in which the first node creates a logical volume matching the storage volume request information via Bdev, includes: Obtain storage volume request information from the CSI controller plugin; Using the CSI node plugin of the first node, a logical volume matching the storage volume request information is created in the storage capacity of the first node via Bdev.

5. The container storage method as described in claim 1, characterized in that, Obtain the second node deployed by the Pod. On the second node, create an initiator service (NVMe-oFInitiator) to connect with the target node through the NVMe-oFInitiator service, which is based on the Ethernet-based Non-Volatile Fast Memory Storage Protocol. This includes: Based on the first node, the Pod is scheduled to obtain the second node where the Pod is deployed; Based on the obtained storage resource information of the target end, configure the NVMe-oF Initiator service parameters and create the initiator through the SPDK API; A connection request is initiated to the target terminal, thereby establishing a storage access channel between the target terminal and the initiating terminal.

6. The container storage method as described in claim 1, characterized in that, If the third node includes the first node, increasing the scoring weight of the first node includes: If the third node includes the first node, the scheduler plugin is invoked via gRPC to increase the scoring weight of the first node; the scheduler plugin is deployed in the extension layer of the Kubernetes scheduler.

7. The container storage method as described in claim 1, characterized in that, The step of generating a local storage interface on the second node through the initiating end includes: Using the CSI node plugin of the second node, a virtual block device is created in the storage capacity of the second node through Bdev, which maps to the storage resources of the target end; A local storage interface is generated based on the virtual block device.

8. A container storage method as described in claim 1, characterized in that, The step of mounting the local storage interface to the Pod's file directory includes: Generate a unique file directory for the Pod on the second node using the Kubernetes Kubelet component; The local storage interface is mounted to the unique file directory via the CSI node plugin.

9. A container storage system, characterized in that, include: The node filtering module is used to select the first node based on the Pod's storage requirements and the storage capacity of each node; The first node is the node that provides storage resources for the Pod; The target creation module is used to create a target in the first node through the NVMe-oF Target service based on the Ethernet-based non-volatile fast memory storage protocol; The initiator creation module is used to obtain the second node where the Pod is deployed. On the second node, an initiator is created to connect with the target node through the NVMe-oF Initiator service based on the Ethernet non-volatile fast storage protocol. A container binding module is used to generate a local storage interface on the second node through the initiating end; the local storage interface is mapped to the storage resources of the target end; The container mounting module is used to mount the local storage interface to the file directory of the Pod; The application data read / write module is used to obtain a first node and a second node when the Pod generates application data. If the first node and the second node are the same, the application data is directly read and written to the target storage resources through SPDK. If the first node and the second node are different, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target for read and write via TCP protocol. The application data read / write module is controlled and scheduled by the container storage system when the Pod's file directory is mounted. Specifically, when the initiating module obtains the second node where the Pod is deployed, it is used to: Obtain the Pod's runtime requirements, and use the Kubernetes scheduler to select a third node from all nodes that meets the Pod's runtime requirements; If the third node includes the first node, increase the scoring weight of the first node; Calculate the operational quality score of the third node based on the scoring weights. Based on the operational quality score of the third node, the second node is selected for Pod scheduling.

10. A container storage device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the container storage method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the container storage method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when executed by a processor, perform the container storage method as described in any one of claims 1 to 8.

Images