Container storage method, system, equipment, medium and product

Abstract

The invention discloses a container storage method, system and device, a medium and a product, and the method comprises the steps: selecting a first node according to a Pod storage demand and the storage capacity of each node; in the first node, creating a target end through a target end service NVMe-oF Target of a nonvolatile fast memory storage protocol based on the Ethernet; acquiring a second node deployed by the Pod, and in the second node, creating an initiating end connected with the target end through an initiating end service NVMe-oF Initiator of a nonvolatile fast memory storage protocol based on the Ethernet; generating a local storage interface at the second node through the initiating end; the local storage interface is mapped to a storage resource of a target end; and mounting the local storage interface on a file directory of the Pod. By adopting the embodiment of the invention, the PCIe local access and the NVMe-oF remote access can be supported at the same time so as to adapt to a multi-container scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of container storage, and in particular to a container storage method, system, device, medium and product. BACKGROUND

[0002] Kubernetes, as an open-source container orchestration platform, can realize efficient management of large-scale container clusters. At the same time, Kubernetes has strong extensibility and supports developers to customize container storage plugins through standardized interfaces to fill the gap between native storage functions and actual business needs.

[0003] However, the current container storage solution has obvious short boards in scene adaptation: for example, the solution based on a file system such as Ceph has performance limitations due to the architecture characteristics of the file system, and it is difficult to adapt to the demand for high-performance storage in the container scenario; for example, the solution of binding node affinity will significantly increase the difficulty of service recovery when the node fails, thereby affecting the overall availability of the service. SUMMARY

[0004] In view of the problems existing in the prior art, the embodiments of the present application provide a container storage method, system, device, medium and product, which can simultaneously support PCIe local access and NVMe-oF remote access to adapt to multi-container scenarios.

[0005] In a first aspect, the embodiments of the present application provide a container storage method, comprising: selecting a first node according to Pod storage requirements and storage capacities of each node; In the first node, a target end of an NVMe-oF Target service based on an Ethernet-based non-volatile fast memory storage protocol is created; obtaining a second node on which a Pod is deployed, and in the second node, an initiator of an NVMe-oF Initiator service based on an Ethernet-based non-volatile fast memory storage protocol is created to connect to the target end; a local storage interface is generated in the second node through the initiator; the local storage interface is mapped to storage resources of the target end; mounting the local storage interface on a file directory of the Pod.

[0006] As an improvement of the above solution, the selecting of the first node according to the Pod storage requirements and the storage capacities of each node comprises: obtaining the storage capacities of each node through a CSI node plugin deployed on each node; obtaining storage capacities of each node according to the storage capacities through a CSI control plugin deployed in the Kubernetes cluster. The Kubernetes scheduler selects a first node from all nodes according to the storage margin and the storage requirement of the Pod.

[0007] As an improvement of the above scheme, the target end in the first node is created by a target end service NVMe-oF Target based on an Ethernet-based non-volatile fast memory storage protocol, and the target end creation includes: Obtaining storage volume request information, the first node creates a logical volume matched with the storage volume request information through Bdev; Binding the logical volume with NVMe-oF transport layer parameters; According to the bound logical volume, initializing the NVMe-oF Target service, exposing the storage service of the logical volume, and generating a target end that can be identified by the initiator; the storage resource of the target end is the logical volume.

[0008] As an improvement of the above scheme, the obtaining of the storage volume request information, and the first node creating a logical volume matched with the storage volume request information through Bdev, includes: Obtaining storage volume request information from a CSI controller plug-in; Using the CSI node plug-in of the first node, creating a logical volume matched with the storage volume request information in the storage capacity of the first node through Bdev.

[0009] As an improvement of the above scheme, the second node where the Pod is deployed is obtained, and in the second node, an initiator connected with the target end is created by an initiator service NVMe-oF Initiator based on an Ethernet-based non-volatile fast memory storage protocol, and the initiator creation includes: Scheduling the Pod based on the first node to obtain the second node where the Pod is deployed; According to the information of the storage resource of the obtained target end, configuring the NVMe-oF Initiator service parameters, and creating the initiator through the SPDK API; Initiating a connection request to the target end to form a storage access channel between the target end and the initiator.

[0010] As an improvement of the above scheme, the scheduling of the Pod based on the first node to obtain the second node where the Pod is deployed includes: Obtaining the running requirement of the Pod, and screening a third node satisfying the running requirement of the Pod from all nodes through the Kubernetes scheduler; If the first node is included in the third node, increasing the scoring weight of the first node; According to the score weight, a running quality score of the third node is calculated; According to the running quality score of the third node, a second node is selected for Pod scheduling.

[0011] As an improvement of the above scheme, if the first node is included in the third node, the score weight of the first node is increased, comprising: If the first node is included in the third node, a scheduler plugin is called through gRPC to increase the score weight of the first node; the scheduler plugin is deployed in an extension layer of a Kubernetes scheduler.

[0012] As an improvement of the above scheme, the local storage interface is generated in the second node through the initiator, comprising: A CSI node plugin of the second node is used to create a virtual block device in the storage capacity of the second node through Bdev, which is mapped with the storage resource of the target end; According to the virtual block device, a local storage interface is generated.

[0013] As an improvement of the above scheme, the local storage interface is mounted on the file directory of the Pod, comprising: A unique file directory of the Pod on the second node is generated through a Kubelet component of Kubernetes; The local storage interface is mounted on the unique file directory through the CSI node plugin.

[0014] As an improvement of the above scheme, if the file directory of the Pod has been mounted, when the Pod generates application data, the first node and the second node are obtained; If the first node and the second node are consistent, the application data is directly read and written to the storage resource of the target end through SPDK; If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into an NVMe command, and the NVMe command is sent to the target end for reading and writing through TCP protocol.

[0015] In a second aspect, an embodiment of the present application further provides a container storage system, comprising: A node screening module is configured to select a first node according to Pod storage requirements and storage capacities of each node; A target end creating module is configured to create a target end in the first node through an NVMe-oF Target of an Ethernet-based non-volatile fast memory storage protocol. The initiator creation module is configured to obtain a second node in which a Pod is deployed, and create, in the second node, an initiator of an Ethernet-based non-volatile fast memory storage protocol (NVMe-oF) initiator service connected to the target through the target. The container binding module is configured to generate a local storage interface in the second node through the initiator; and the local storage interface is mapped to a storage resource of the target. The container mounting module is configured to mount the local storage interface on a file directory of the Pod.

[0016] In a third aspect, an embodiment of the present application provides a container storage device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor executes the computer program to implement the container storage method as described above.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, including a stored computer program, and when the computer program runs, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the container storage method as described above.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the container storage method as described above is executed.

[0019] Compared with the prior art, the container storage method, system, device, medium and product provided by the present application can select a first node according to a storage requirement of a Pod and storage capacities of nodes; in the first node, a target of an Ethernet-based non-volatile fast memory storage protocol (NVMe-oF) target service is created; a second node in which a Pod is deployed is obtained, and in the second node, an initiator of the Ethernet-based non-volatile fast memory storage protocol (NVMe-oF) initiator service is created to be connected to the target; a local storage interface is generated in the second node through the initiator; the local storage interface is mapped to a storage resource of the target; and the local storage interface is mounted on a file directory of the Pod. The embodiment of the present application can support both PCIe local access and NVMe-oF remote access to adapt to a multi-container scenario. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a step flowchart of a container storage method provided by an embodiment of the present application; Figure 2is a schematic diagram of a capacity storage plug-in provided by an embodiment of the present application; Figure 3 is a scheduling flowchart at the level of a container storage plug-in provided by an embodiment of the present application; Figure 4 is a step flowchart of a Pod using a mounted data volume provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of a container storage system provided by an embodiment of the present application; Figure 6 is a structural block diagram of a container storage device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0022] In the description and claims of the specification, it is to be understood that the terms first, second, etc. are used only for the purpose of description and are not to be construed as indicating or implying relative importance or an indicated number of technical features. They are not necessarily used in the order or time sequence described. Where appropriate, terms are interchangeable. Thus, features with "first" and "second" designations can implicitly or explicitly include at least one of the features.

[0023] The native Kubernetes platform manages and controls a plurality of nodes in the cluster server through components such as an API Server (application program interface service), a Scheduler, a Controller Manager, and a Kubelet (node agent). Among them, the API Server is responsible for receiving and processing all resource operation requests, such as a Pod creation request and a storage creation request; the Scheduler is responsible for binding unscheduled Pods to appropriate nodes; the Controller Manager is responsible for managing various controllers; and the Kubelet is deployed in each node to take over the full life cycle management of the node Pods.

[0024] The Kubernetes native components build the basic management and control capabilities of the cluster server, but there is still a gap between them and the actual business needs, which needs to be filled by developing container storage plug-ins. The current container storage plug-ins mainly include adaptive plug-ins and cloud-native plug-ins.

[0025] The adaptive plug-in is adapted to the distributed file system by accessing the container storage specification (Container Storage Interface, CSI). For example, ceph-csi based on ceph, and gluster-heketi-csi-driver based on Glusterfs. The cloud-native plug-in is for the container scenario, and a dedicated storage architecture is constructed around the container demand. For example, Longhorn based on the iSCSI protocol, which realizes the management of the storage volume on the control plane by combining the container specification with the self-developed system, and realizes the cross-node data writing on the data plane through the iSCSI protocol. For another example, OpenEBS adds node affinity to the data volume through lvm, hostpath and the like, thereby providing efficient and simple local storage.

[0026] The performance of the existing adaptive plug-in is limited by the file system itself. For example, the cloud-native plug-in of Longhorn is also limited by the performance of the iSCSI technology, and cannot meet the performance requirements of high-performance storage in the container scenario. Although the cloud-native plug-in based on the hostpath mode can meet the high-performance requirements, the node is bound with affinity, which significantly increases the difficulty of service recovery when the node fails, thereby affecting the overall availability of the service.

[0027] The current container storage solution has more or less its own problems, and cannot provide a complete high-performance storage solution suitable for multiple container scenarios.

[0028] Based on the above thinking, the embodiment of the present application provides a container storage method. Please refer to Figure 1 In this embodiment, the container storage method is specifically executed through steps S1 to S5. S1, selecting a first node according to the storage demand of a Pod and the storage capacity of each node.

[0029] S2, in the first node, creating a target end through a target end service NVMe-oF Target based on an Ethernet-based non-volatile fast memory storage protocol.

[0030] S3, obtaining a second node on which the Pod is deployed, and creating an initiator in the second node through a initiator service NVMe-oF Initiator based on an Ethernet-based non-volatile fast memory storage protocol, the initiator being connected to the target end.

[0031] S4, generating a local storage interface in the second node through the initiator; the local storage interface is mapped to the storage resource of the target end.

[0032] S5, mounting the local storage interface on the file directory of the Pod.

[0033] It should be noted that the file directory of the Pod is mounted, that is, the Pod is associated with a certain storage resource, so that the Pod can persistently read and write data through the directory.

[0034] NVMe-oF (Non-Volatile Memory Express over Fabrics, Ethernet-based non-volatile fast memory storage protocol) technology is a remote storage technology extended on the basis of NVM storage protocol, which extends local NVMe to remote storage devices through Fabric network, so that remote nodes can access remote storage resources with low delay and high bandwidth as if accessing local PCIe SSD.

[0035] The core role of the target service NVMe-oF Target of the Ethernet-based non-volatile fast memory storage protocol is to encapsulate the physical storage on the node as a target Target that can be shared through the NVMe-oF network, and receive access requests from other nodes. The core role of the initiator service NVMe-oF Initiator of the Ethernet-based non-volatile fast memory storage protocol is to actively initiate a connection request to the NVMe-oF Target to establish a remote storage channel and identify the remote storage provided by the target Target as a locally accessible storage device.

[0036] Preferably, NVMe-oF is based on SPDK (Storage Performance Development Kit, high-performance storage development kit).

[0037] It should be noted that the first node is a node providing storage resources for the Pod, the second node is a node actually deploying and running the Pod, and is also a node where the storage resource access end is located, and the storage requirement of the Pod is realized by interaction between the first node and the second node.

[0038] In some preferred embodiments, the first node and the second node are the same node in the cluster server, and the interaction between the first node and the second node is realized through a PCIe local channel; in other preferred embodiments, the first node and the second node are different nodes in the cluster server, and the interaction between the nodes is realized through an NVMe-oF remote channel, and the entity storage volume of the Pod resource is located in the first node.

[0039] In the embodiments of the present application, the NVMe over Fabrics technology is adopted to provide the capability of container cross-node storage sharing, on the basis of which, PCIe local channel and NVMe-oF remote channel can be simultaneously supported, so that the data disk on the cluster server can be directly accessed by local applications, and can also be accessed by the remaining nodes through the network, to adapt to the multi-container scenario.

[0040] Preferably, NVMe-oF is connected to Kubernetes through the CSI (Container Storage Interface) plug-in to provide high-speed persistent storage for Kubernetes containers.

[0041] It should be noted that CSI is a standardized specification currently used in the industry. Its goal is to unify the docking interface between the container orchestration platform and the underlying storage system. It defines a set of common APIs for operations such as creating and mounting volumes. By developing plug-ins that comply with the CSI standard, it can be adapted to all container platforms that support CSI.

[0042] Preferably, a CSI control plug-in and a CSI node plug-in are constructed in the embodiment of the present invention.

[0043] The CSI control plug-in implements the Controller portion of the container storage plug-in specification and is responsible for the pre-container creation process, including metadata management and calling the plug-in on the application node. It should be noted that only one replica in the cluster can provide services at any one time.

[0044] The CSI node plug-in is deployed on each node, implementing the Node part of the container storage plug-in specification. It is responsible for the actual creation of storage and binding to Pods, and provides an interface for CSI control plug-ins to call.

[0045] See Figure 2 The Container Storage Interface (CSI) control plug-in can call the Container Storage Interface (CSI) node plug-in of the application node to create a High-Performance Storage Development Kit (SPDK) target or initiator. The CSI node plug-in can create a virtual block through the SPDK block device layer (Bdev), and then create a target based on the SPDK NVMe-oF Target service or an initiator based on the SPDK NVMe-oF Initiator service.

[0046] As a preferred implementation, step S1, selecting a first node based on the Pod storage requirements and the storage capacity of each node, includes: Obtain the storage capacity of each node through the CSI node plug-in deployed on each node; The CSI control plug-in deployed in the Kubernetes cluster obtains the storage margin of each node based on the storage capacity. The Kubernetes scheduler selects the first node from all nodes based on the storage margin and the Pod storage requirement.

[0047] In actual application, the CSI node plug-in of each node continuously collects and reports the storage capacity of the node itself, the CSI node plug-in runs locally on the node and can perceive the change of the storage capacity with low delay; the CSI control plug-in aggregates the storage capacity of all nodes and judges which nodes have storage capacity, and further feeds back to the Kubernetes. In the volume scheduling stage, the Kubernetes scheduler selects the first node from the candidate list provided by the CSI control plug-in in combination with multi-dimensional constraints.

[0048] As a preferred embodiment, step S2, in the first node, a target end is created by an Ethernet-based non-volatile fast memory storage protocol target end service NVMe-oF Target, including: Obtaining storage volume request information, the first node creates a logical volume matched with the storage volume request information through Bdev; Binding the logical volume with NVMe-oF transport layer parameters; Initializing the NVMe-oF Target service according to the bound logical volume, exposing the storage service of the logical volume, and generating a target end that can be identified by an initiator; the storage resource of the target end is the logical volume.

[0049] Preferably, the storage volume request information includes first node information, storage volume type information and / or storage volume scale information.

[0050] Further, preferably, the obtaining of the storage volume request information, the first node creates a logical volume matched with the storage volume request information through Bdev, includes: Obtaining the storage volume request information from the CSI controller plug-in; Using the CSI node plug-in of the first node, a logical volume matched with the storage volume request information is created in the storage capacity of the first node through Bdev.

[0051] In the embodiment of the application, after the first node is selected, the CSI control plug-in distributes the storage volume request information to the CSI node plug-in of the first node. It should be noted that the CSI control plug-in does not directly operate the node hardware.

[0052] Then the CSI node plug-in of the first node efficiently creates a logical volume matched with the storage volume request information through the core component Bdev (Block Device) of the SPDK framework. In the embodiment of the application, the logical volume is an independent block device partition divided on a physical NVMe SSD. Through the Bdev of SPDK, the switching overhead of the kernel mode and the user mode can be reduced, and the IO performance can be improved.

[0053] The transport layer parameter is a communication address of the NVMe-oF, and defines a network path for the Target service to provide access externally. By binding the logical volume with the transport layer parameter, the reachability of cross-border access is ensured.

[0054] Initializing the NVMe-oF Target service is to encapsulate the logical volume bound with the transport layer parameter into a remote block device service conforming to the NVMe-oF protocol to generate a target end, so that the subsequent initiator can accurately identify and connect to the target end.

[0055] As a preferred embodiment, the step S3 of obtaining the second node deployed by the Pod, in which the initiator is created to connect to the target end by the NVMe-oF initiator based on the Ethernet-based non-volatile fast memory storage protocol, comprises the following steps: Based on the first node, the Pod is scheduled to obtain the second node deployed by the Pod; According to the information of the storage resource of the target end obtained, the NVMe-oF initiator service parameter is configured, and the initiator is created through the SPDK API; A connection request is initiated to the target end to form a storage access channel between the target end and the initiator.

[0056] After the volume is created through the API provided by the CSI specification in step S2, the Pod needs to be scheduled, and the Pod scheduling phase is entered. In the traditional scheduling process, the Pod scheduling is performed by the Kubernetes scheduler to comprehensively consider the running quality of each node, select the second node, and deploy and run the Pod on the second node.

[0057] It should be noted that in the preferred embodiment, in the Pod scheduling phase, the first node selected is considered, and the first node is exemplarily provided with a special identifier, so that the Kubernetes scheduler preferentially considers the first node or a node in the same network segment and the same rack as the first node as the second node, thereby reducing the network delay of the NVMe-oF remote channel.

[0058] The CSI control plug-in sends the information of the storage resource of the target end to the CSI node plug-in of the second node, so that the CSI node plug-in creates the initiator through the NVMe-oF initiator service based on the SPDK based on the information.

[0059] By sending the NVMe-oF connection request from the initiator to the target end, the bidirectional authentication and channel establishment of the first node Target to the second node Initiator can be completed, so that the second node has a physical channel to access the storage resource of the first node.

[0060] Further, preferably, the second node to which the pod is deployed is obtained based on the first node, comprising: obtaining a running requirement of the pod, and filtering, by a Kubernetes scheduler, third nodes from all nodes to meet the running requirement of the pod; if the first node is included in the third nodes, increasing a score weight of the first node; calculating a running quality score of the third nodes according to the score weight; selecting the second node to schedule the pod according to the running quality score of the third nodes.

[0061] Based on the NVMe-oF technology, the pod running node and the storage node can be separated, in the preferred embodiment of the present application, considering that the data transmission in the same node has higher real-time performance than the data transmission between different nodes, the first node selected in the pod scheduling stage is further considered to make the running service and the storage service of the pod fall on the same node.

[0062] In some preferred embodiments, the embodiment of the present application further constructs a scheduler plugin calling through gRPC, the scheduler plugin is an extended plugin of the Kubernetes scheduler, and is responsible for preferentially selecting the node where the storage is located during the Kubernetes scheduling. Specifically, the score flow is extended to the score of the node containing the storage node, thereby affecting the priority of node selection.

[0063] Through the extended scheduler plugin, the local priority scheduling algorithm is used in the pod scheduling stage, so that the storage volume and the pod node are as far as possible on the same node, so that the data can be directly written to the local data disk without crossing the network, so as to shorten the data read-write path.

[0064] Further, preferably, if the first node is included in the third nodes, the score weight of the first node is increased, comprising: if the first node is included in the third nodes, a scheduler plugin is called through gRPC to increase the score weight of the first node; the scheduler plugin is deployed in an extension layer of the Kubernetes scheduler.

[0065] In some preferred embodiments, the running logic of the scheduler plugin is that when the Kubernetes scheduler scores, the weight of all nodes of the volume where the first node is located is added according to the information of the first node provided by the CSI control plugin, and the weight of each node is determined based on the proximity of the node to the first node.

[0066] As a preferred implementation, step S4, generating a local storage interface at the second node by the initiator, includes: Using the CSI node plug-in of the second node, creating a virtual block device in the storage capacity of the second node through Bdev, which is mapped to the storage resource of the target end; A local storage interface is generated according to the virtual block device.

[0067] It's important to note that the virtual block device created by the second node through Bdev is a logical mapping of the first node's storage resources. It occupies the second node's device node identifier and does not consume the second node's local persistent storage. Through the mapping relationship between the virtual block device and the storage resource, the stored data is stored on the first node's storage resources.

[0068] By generating a local storage interface, the Pod can identify and mount SPDK internal objects. The Pod does not need to know whether the storage is a local PCIe disk or a remote NVMe-oF volume, and can access it through a standard path.

[0069] As a preferred implementation, step S5, mounting the local storage interface on the file directory of the Pod, includes: Generate a unique file directory for the Pod on the second node through the Kubelet component of Kubernetes; The local storage interface is mounted on the unique file directory through the CSI node plug-in.

[0070] Multiple Pods may run simultaneously in the second node. The Kubelet component creates a unique file directory for the Pod and sets access permissions for the Pod, thereby coordinating the storage data of each Pod. In addition, as a native component responsible for Pod lifecycle management, the Kubelet component can also coordinate the interaction timing of components such as the CSI plug-in to ensure the orderly progress of the container storage process.

[0071] In the above solution, by mounting the local storage interface to a unique file directory, the applications within the Pod are exposed to a standard file system directory. There is no need to be aware of details such as the data is actually stored on the remote node or that the underlying storage layer uses the SPDK-based NVMe-oF protocol. Data can be written to remote storage and read from remote data simply by operating the system like reading and writing a local directory, which reduces the difficulty of developing and maintaining Pod applications.

[0072] In some preferred embodiments, when a Pod is deleted, the Kubelet component triggers the corresponding CSI plugin to perform an unmounting operation and clean up the unique file directory. The embodiments of the present application can automatically realize mounting and unmounting with Pod start and stop, ensure that storage resources are synchronized with the Pod life cycle, and improve cluster resource utilization.

[0073] Please refer to Figure 3 , Figure 3 A container storage plugin level scheduling flowchart is provided based on the CSI specification.

[0074] When the application interface service API Server receives a PVC (Persistent Volume Claim) and a Pod submitted by a user, the volume scheduling stage is entered, the PVC is listened to by the Kubernetes native controller manager Controller Manager, and it is determined whether the PVC has a corresponding PV (Persistent Volume).

[0075] If there is no corresponding PV, the volume creation stage is entered, the PVC event is listened to by the resource automatic allocator External Provisioner, the volume creation instruction is issued to the CSI control plugin created by the embodiments of the present application, the SPDK Target target end is created by the CSI control plugin coordinating the CSI node plugin of the first node. The CSI node plugin abstracts the physical NVMe disk into a block device that can be managed by SPDK through a virtual block Bdev, configures an NVMe TCP Transport transport layer, enables NVMe-oF network transmission capability, and then registers the target end SPDK Target to provide NVMe-oF storage services externally. After the volume creation is successful, the application interface service API Server is notified by the resource automatic allocator External Provisioner to create a PV, so as to complete the binding of the PVC and the PV.

[0076] After the volume creation stage ends, the Pod scheduling stage is entered. The information of the SPDK storage volume is read by the Kubernetes scheduler to score all candidate nodes. The weight score of the first node is adjusted by calling the scheduler plugin created by the embodiments of the present application in the scoring process. Finally, the scored nodes are returned to the Kubernetes scheduler to determine the second node where the Pod is located.

[0077] When the node where the Pod is located is determined, the volume binding phase can be entered. The CSI node plugin of the second node creates a client SPDK Initiator, then establishes a remote storage channel through the NVMe-oF protocol, connects the target SPDK Target of the first node, and then creates a virtual block device locally and maps it to the corresponding remote real volume storage. At this time, the second node can access the storage resources of the first node through the client SPDK Initiator.

[0078] When the virtual block device and the Pod are both ready, the CSI node plugin can bind the virtual block device and the Pod. In the volume mounting phase, Kubernetes prepares a unique file directory for the Pod for mounting on the virtual block device, and then the Pod can correctly use the corresponding block storage.

[0079] The complete PVC and Pod creation process is completed through the above steps S1-S5. After that, the Pod can be normally started and use the mounted data volume. Subsequent data reading and writing of the file system will be circulated through SPDK.

[0080] As a preferred embodiment, if the file directory of the Pod has been mounted by the above embodiment, the container storage method specifically executes steps S6 to S8: S6, when the Pod generates application data, the first node and the second node are obtained; S7, if the first node and the second node are consistent, the application data is directly read and written to the target storage resource through SPDK; S8, if the first node and the second node are inconsistent, the application data received by the local storage interface is converted into an NVMe command, and the NVMe command is sent to the target for reading and writing through the TCP protocol.

[0081] Preferably, the NVMe command includes an NVMe storage command and an NVMe read command.

[0082] Taking data writing as an example, there are two cases according to the consistency of the Pod and the storage node.

[0083] In most cases, the Pod and the storage are on the same node, and at this time the data can be directly written to the created logical data disk through SPDK without passing through the TCP / IP protocol stack and without network loss.

[0084] In some cases, the Pod and the storage are not on the same node, for example, the node CPU pressure is large, and the application is evicted to other nodes, at this time data transmission needs to be performed across the network. Please refer to Figure 4After the application data of the Pod is written into the virtual block device through the file system, the data of the virtual block storage is converted into an NVMe storage command, and then sent to the remote NVMe target through the TCP protocol, and the target end target writes the storage command into the associated real storage medium after receiving the storage command.

[0085] The container storage method provided by the embodiment of the application adopts the NVMe over Fabrics technology, provides the capability of container cross-node storage sharing, and can adapt to a multi-container scenario, and on this basis, can simultaneously support a PCIe local channel and an NVMe-oF remote channel, so that the data disk on the cluster server can be directly accessed by a local application and can be accessed by the rest of the nodes through a network, and through the design of separated computing and storage, it is ensured that the node failure and other problems can be quickly self-healed.

[0086] The embodiment of the application provides a container storage system. Figure 5 The container storage system controls and schedules a node screening module 11, a target end creation module 12, an initiator end creation module 13, a container binding module 14 and a container mounting module 15 therein, and wherein: The node screening module 11 is configured to select a first node according to a Pod storage requirement and a storage capacity of each node. The target end creation module 12 is configured to create a target end of an NVMe-oF Target service based on an Ethernet-based nonvolatile fast memory storage protocol in the first node. The initiator end creation module 13 is configured to obtain a second node on which a Pod is deployed, and create an initiator end of an NVMe-oF Initiator service based on an Ethernet-based nonvolatile fast memory storage protocol in the second node, so as to connect the target end. The container binding module 14 is configured to generate a local storage interface in the second node through the initiator end. The container mounting module 15 is configured to mount the local storage interface on a file directory of the Pod.

[0087] As a preferred embodiment, the node screening module 11 is specifically configured to: Obtain the storage capacity of each node through a CSI node plug-in deployed on each node. Obtain the storage capacity of each node through a CSI control plug-in deployed on the Kubernetes cluster. The Kubernetes scheduler selects the first node from all nodes according to the storage margin and the storage requirement of the Pod.

[0088] As a preferred implementation, the target end creation module 12 comprises: a logical volume creation unit configured to obtain storage volume request information, and create, by the first node, a logical volume matching the storage volume request information through Bdev; a transport layer parameter binding unit configured to bind the logical volume with NVMe-oF transport layer parameters; a target end generation unit configured to initialize an NVMe-oF Target service according to the bound logical volume, expose a storage service of the logical volume, and generate a target end identifiable by an initiator; and the storage resource of the target end is the logical volume.

[0089] Further, preferably, the logical volume creation unit is specifically configured to: obtain the storage volume request information from a CSI controller plugin; create, by the CSI node plugin of the first node, a logical volume matching the storage volume request information in the storage capacity of the first node through Bdev.

[0090] As a preferred implementation, the initiator creation module 13 comprises: a Pod scheduling unit configured to schedule a Pod based on the first node to obtain a second node to which the Pod is deployed; an initiator creation unit configured to configure NVMe-oF Initiator service parameters according to the obtained information of the storage resource of the target end, and create an initiator through an SPDK API; a connection request unit configured to initiate a connection request to the target end to form a storage access channel between the target end and the initiator.

[0091] Further, preferably, the Pod scheduling unit is specifically configured to: obtain a Pod running requirement, and filter a third node satisfying the Pod running requirement from all nodes through a Kubernetes scheduler; if the first node is included in the third node, increase a scoring weight of the first node; calculate a running quality score of the third node according to the scoring weight; select the second node for Pod scheduling according to the running quality score of the third node.

[0092] Further, preferably, if the first node is included in the third node, the score weight of the first node is increased, comprising: If the first node is included in the third node, a scheduler plugin is called through gRPC to increase the score weight of the first node; the scheduler plugin is deployed in an extension layer of a Kubernetes scheduler.

[0093] As a preferred embodiment, the container binding module 14 is specifically used for: A virtual block device mapped with the storage resource of the target end is created in the storage capacity of the second node through Bdev by using the CSI node plugin of the second node; A local storage interface is generated according to the virtual block device.

[0094] As a preferred embodiment, the container mounting module 15 is specifically used for: A unique file directory of a Pod on the second node is generated through a Kubelet component of Kubernetes; The local storage interface is mounted on the unique file directory through the CSI node plugin.

[0095] As a preferred embodiment, if the file directory of the Pod is mounted, the container storage system further controls and schedules an application data read-write module therein, and the application data read-write module is used for: When application data of the Pod is generated, the first node and the second node are acquired; If the first node and the second node are consistent, the application data is directly read and written to the storage resource of the target end through SPDK; If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into an NVMe command, and the NVMe command is sent to the target end for reading and writing through a TCP protocol.

[0096] By using the container storage system provided by the embodiment of the application, the NVMe over Fabrics technology is adopted to provide the capability of container cross-node storage sharing, and can adapt to a multi-container scenario, and on this basis, can simultaneously support a PCIe local channel and an NVMe-oF remote channel, so that the data disk on the cluster server can be directly accessed by a local application and can be accessed by the rest of the nodes through a network, and through the design of computing and storage separation, it is ensured that the file system can be quickly self-healed after problems such as node failure occur.

[0097] Referring to Figure 6 , Figure 6is a structural block diagram of a container storage device provided by an embodiment of the present application. The container storage device comprises a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. The processor 31 implements the steps in each of the above container storage method embodiments when executing the computer program, such as steps S1-S5 and / or S6-S8.

[0098] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the container storage device.

[0099] The container storage device can include, but is not limited to, the processor 31 and the memory 32. Those skilled in the art can understand that the schematic diagram is only an example of the container storage device and does not constitute a limitation on the container storage device, which can include more or fewer components than the diagram, or combine certain components, or different components, for example, the container storage device can also include an input / output device, a network access device, a bus, etc.

[0100] The processor 31 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 31 is the control center of the container storage device, which connects various parts of the container storage device through various interfaces and lines.

[0101] The memory 32 can be used to store the computer programs and / or modules, and the processor 31 realizes various functions of the container storage device by running or executing the computer programs and / or modules stored in the memory 32, and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0102] The modules / units integrated in the container storage device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor 31 executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0103] The above is the preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements are also considered within the protection scope of the present application.

Claims

1. A container storage method, characterized in that: include: Select the first node based on the Pod storage requirements and the storage capacity of each node; In the first node, a target end is created through a target end service NVMe-oF Target of a non-volatile memory express storage protocol over Ethernet; Obtain a second node where the Pod is deployed, and in the second node, create an initiator connected to the target end through an initiator service NVMe-oF Initiator of a non-volatile memory over Ethernet protocol; Generate a local storage interface on the second node through the initiator; map the local storage interface to the storage resource of the target end; Mount the local storage interface on the file directory of the Pod.

2. A container storage method according to claim 1, characterized in that: The step of selecting the first node based on the Pod storage requirements and the storage capacity of each node includes: Obtain the storage capacity of each node through the CSI node plug-in deployed on each node; The CSI control plug-in deployed in the Kubernetes cluster obtains the storage margin of each node based on the storage capacity. The Kubernetes scheduler selects the first node from all nodes based on the storage margin and the Pod storage requirement.

3. A container storage method according to claim 1, characterized in that: The step of creating a target end in the first node by using a target end service NVMe-oF Target of a non-volatile memory express storage protocol over Ethernet includes: Acquire storage volume request information, and the first node creates a logical volume matching the storage volume request information through Bdev; Binding the logical volume to the NVMe-oF transport layer parameters; According to the bound logical volume, the NVMe-oF Target service is initialized, the storage service of the logical volume is exposed, and a target end that can be identified by the initiator is generated; the storage resource of the target end is the logical volume.

4. A container storage method according to claim 3, characterized in that: The acquiring of the storage volume request information, wherein the first node creates a logical volume matching the storage volume request information through Bdev, includes: Get storage volume request information from the CSI controller plug-in; The CSI node plug-in of the first node is used to create a logical volume matching the storage volume request information in the storage capacity of the first node through Bdev.

5. A container storage method according to claim 1, characterized in that: Obtain a second node where the Pod is deployed, and in the second node, create an initiator connected to the target end through an initiator service NVMe-oFInitiator of a non-volatile memory express storage protocol based on Ethernet, including: Based on the first node, the Pod is scheduled to obtain a second node where the Pod is deployed; Configure the NVMe-oF Initiator service parameters based on the target storage resource information obtained, and create an initiator through the SPDK API. A connection request is initiated to the target end to form a storage access channel between the target end and the initiating end.

6. A container storage method according to claim 5, characterized in that: Scheduling the Pod based on the first node to obtain the second node where the Pod is deployed includes: Obtain the Pod running requirements and use the Kubernetes scheduler to select a third node that meets the Pod running requirements from all nodes; If the third node includes the first node, increase the scoring weight of the first node; Calculating the operation quality score of the third node according to the scoring weight; According to the operation quality score of the third node, the second node is selected for Pod scheduling.

7. A container storage method according to claim 6, characterized in that: If the third node includes the first node, increasing the scoring weight of the first node includes: If the third node includes the first node, a scheduler plug-in is called through gRPC to increase the scoring weight of the first node; the scheduler plug-in is deployed in the extension layer of the Kubernetes scheduler.

8. A container storage method according to claim 1, characterized in that: Generating a local storage interface on the second node by the initiator includes: Using the CSI node plug-in of the second node, creating a virtual block device in the storage capacity of the second node through Bdev, which is mapped to the storage resource of the target end; A local storage interface is generated according to the virtual block device.

9. A container storage method according to claim 1, characterized in that: Mounting the local storage interface on the file directory of the Pod includes: Generate a unique file directory for the Pod on the second node through the Kubelet component of Kubernetes; The local storage interface is mounted on the unique file directory through the CSI node plug-in.

10. A container storage method according to claim 1, characterized in that: If the Pod's file directory is mounted, when the Pod generates application data, the first and second nodes are obtained; If the first node and the second node are consistent, the application data is directly read and written to the storage resources of the target end through SPDK; If the first node and the second node are inconsistent, the application data received by the local storage interface is converted into NVMe commands, and then the NVMe commands are sent to the target end for reading and writing through the TCP protocol.

11. A container storage system, characterized in that: include: The node screening module is used to select the first node based on the Pod storage requirements and the storage capacity of each node; A target end creation module is used to create a target end in the first node through the target end service NVMe-oF Target of the non-volatile memory over Ethernet storage protocol; An initiator creation module is used to obtain a second node where the Pod is deployed, and in the second node, create an initiator connected to the target end through an initiator service NVMe-oF Initiator based on Ethernet non-volatile memory storage protocol; A container binding module, configured to generate a local storage interface on the second node through the initiator; the local storage interface is mapped to the storage resource of the target end; The container mounting module is used to mount the local storage interface on the file directory of the Pod.

12. A container storage device, characterized in that: The container storage method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the container storage method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the container storage method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or the computer instructions are executed by a processor, the container storage method according to any one of claims 1 to 10 is executed.

Images