Container-based cloud hard disk management method and device

Abstract

The application relates to a container-based cloud hard disk management method and device, which comprises the following steps: a ControllerServer component of a CSI plug-in communicates with an ESSD cluster based on a gRPC protocol and manages an ESSD cloud hard disk; an nvmf_tgt service of SPDK is run on a host computer, interacts with the ESSD cloud hard disk in the ESSD cluster through an NVMe-oF protocol and an RDMA network, and creates a block device BDEV corresponding to the ESSD cloud hard disk locally; a SPDK socket file spdk.sock and a directory where a SPDK remote procedure call script rpc.py are located on the host computer are mounted into a Pod running a NodeServer component of the CSI plug-in; the rpc.py script is called in the Pod through the NodeServer component, and the nvmf_tgt service of the host computer is interacted with based on a spdk-rpc command to perform mounting or unmounting operations on the block device BDEV. Through integration of SPDK, NVMe-of and RDMA, the kernel bottleneck can be broken through, and through the mounting of the sock file, the problem of interaction between the CSI node component and the host service is well solved, and finally the containerized efficient use of the high-performance cloud hard disk is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of containers, in particular to a container-based cloud hard disk management method and device. BACKGROUND

[0002] With the rapid development of cloud computing and container technology, Kubernetes has become a de facto standard in the field of container orchestration. The establishment of its Container Storage Interface (CSI) specification aims to standardize the integration between container orchestration systems and storage backends, enabling storage vendors to provide persistent storage services for containerized applications through a unified interface. In this context, how to provide low-latency, high-IOPS (Input / Output Operations Per Second) storage support for high-performance applications running in Kubernetes has become an important technical challenge.

[0003] Currently, common solutions rely on general distributed storage systems such as Ceph, and provide storage services through their CSI plugins. Major cloud vendors (such as Huawei and Alibaba Cloud) also provide their own CSI plugins to integrate their cloud storage products. However, these existing technical solutions have obvious limitations: traditional distributed storage systems such as Ceph have bottlenecks in I / O performance due to their architecture and network protocols, making it difficult to meet the requirements of extreme performance scenarios; while the CSI plugins of major cloud vendors can interface with their own storage services, they are usually based on standard kernel drivers and general network protocols (such as iSCSI) and do not optimize high-performance underlying technologies such as NVMe-oF and RDMA (Remote Direct Memory Access), which cannot fully utilize the potential of new-generation high-performance cloud hard disks. Therefore, there is a lack of an efficient solution to bridge Kubernetes container platforms and underlying high-performance cloud hard disks. SUMMARY

[0004] The present application provides a container-based cloud hard disk management method and device.

[0005] According to a first aspect of the present application, a container-based cloud hard disk management method is provided, which includes:

[0006] Through the ControllerServer component of the CSI plugin, communicate with the ESSD cluster based on the gRPC protocol, and manage the ESSD cloud hard disk in response to the persistent volume claim PVC request from Kubernetes;

[0007] running an nvmf_tgt service of SPDK on a host machine, interacting with an ESSD cloud hard disk in the ESSD cluster through an NVMe-oF protocol and an RDMA network, to locally create a block device BDEV corresponding to the ESSD cloud hard disk;

[0008] mounting a SPDK socket file spdk.sock and a directory where a SPDK remote procedure call script rpc.py is located on the host machine, to a Pod where a NodeServer component running the CSI plugin is located;

[0009] when receiving a mounting or unmounting request of the Kubernetes to the ESSD cloud hard disk, calling the rpc.py script in the Pod through the NodeServer component, and interacting with the nvmf_tgt service of the host machine based on a spdk-rpc command, to perform a mounting or unmounting operation on the block device BDEV.

[0010] According to a second aspect of the embodiment of the present application, a container-based cloud hard disk management apparatus is provided, and the apparatus comprises:

[0011] a management module configured to communicate with an ESSD cluster based on a gRPC protocol through a ControllerServer component of a CSI plugin, and manage an ESSD cloud hard disk in response to a persistent volume claim PVC request from a Kubernetes;

[0012] an interaction module configured to run an nvmf_tgt service of SPDK on a host machine, interact with an ESSD cloud hard disk in the ESSD cluster through an NVMe-oF protocol and an RDMA network, to locally create a block device BDEV corresponding to the ESSD cloud hard disk;

[0013] a mounting module configured to mount a SPDK socket file spdk.sock and a directory where a SPDK remote procedure call script rpc.py is located on the host machine, to a Pod where a NodeServer component running the CSI plugin is located;

[0014] a processing module configured to, when receiving a mounting or unmounting request of the Kubernetes to the ESSD cloud hard disk, call the rpc.py script in the Pod through the NodeServer component, and interact with the nvmf_tgt service of the host machine based on a spdk-rpc command, to perform a mounting or unmounting operation on the BDEV (Block Device).

[0015] According to a third aspect of the embodiments of the present application, an electronic device is provided. The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described above when executing the program.

[0016] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method described above.

[0017] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the method described above.

[0018] The container-based cloud hard disk management method and device provided by the embodiments of the present application can get rid of the kernel bottleneck by integrating SPDK, NVMe-of and RDMA technology, and can solve the problem of interaction between the CSI node component and the host service by mounting the sock file, so as to realize efficient use of the containerized high-performance cloud hard disk. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the following description of exemplary embodiments in conjunction with the accompanying drawings, more details, features and advantages of the present application are disclosed, in which:

[0020] Figure 1 The architecture schematic diagram of the provisioner component provided for an exemplary embodiment of the present application is shown in the figure;

[0021] Figure 2 The JSON file writing flowchart provided for an exemplary embodiment of the present application is shown in the figure;

[0022] Figure 3 The JSON file reading flowchart provided for an exemplary embodiment of the present application is shown in the figure;

[0023] Figure 4 The file system mounting flowchart of the node server provided for an exemplary embodiment of the present application is shown in the figure;

[0024] Figure 5 The flowchart of the container-based cloud hard disk management method provided for an exemplary embodiment of the present application is shown in the figure;

[0025] Figure 6 The functional module schematic block diagram of the container-based cloud hard disk management device provided for an exemplary embodiment of the present application is shown in the figure;

[0026] Figure 7 The structure block diagram of the electronic device provided for an exemplary embodiment of the present application is shown in the figure;

[0027] Figure 8 A structural block diagram of a computer system provided for an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] Embodiments of the present application will be described in more detail by referring to the drawings. Although some embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein, but rather the embodiments are provided so that the present application can be more thoroughly and completely understood. It is understood that the drawings and embodiments of the present application are for exemplary purposes only, and are not intended to limit the scope of protection of the present application.

[0029] It is understood that each of the steps recited in the method embodiments of the present application can be executed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0030] The term “comprising” and variations thereof as used herein are open-ended, that is, “including but not limited to”. The term “based on” is “based, at least in part, on”. The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Related terms are defined as follows. It is to be noted that the terms “first”, “second”, and the like used in the present application are merely used to distinguish different devices, modules, or units, and do not imply the order or interdependence of the functions performed by these devices, modules, or units.

[0031] It is to be noted that the modification of “one” or “multiple” mentioned in the present application is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as “one or more”.

[0032] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] In order to solve the technical problems existing in the related art, the embodiments of the present application provide a container-based cloud hard disk management scheme, which can avoid the overhead of the traditional storage stack, and can directly use NVMe-oF (NVMe over Fabrics, NVMe storage access protocol based on network) and RDMA and other technologies to realize high-performance and low-latency storage access, while maintaining the standardization and ease of use of the CSI standard.

[0034] The embodiment of the application can be used in a cloud computing processing platform and is deployed on a physical machine to run. In order to support interaction with an ESSD (Enhanced Solid State Drive), the embodiment is designed for two main components, a Provisioner and a Node plugin. The Provisioner component is responsible for receiving requests for creation, deletion and the like of a PVC (Persistent Volume Claim) and calling a backend storage system (namely, an ESSD) to manage the life cycle of a volume. The Node plugin is responsible for completing operations such as mounting or unmounting a volume on a Kubernetes node.

[0035] (1) Design of the provisioner architecture.

[0036] The main component in the provisioner is a ControllerServer, which is responsible for operations of the control plane, such as creation, deletion, expansion or viewing of volume information, and the like, which depend on a backend ESSD cluster.

[0037] Because the ESSD cluster environment is complex, and there is no mature client library on the market like ceph, the embodiment uses grpc (Google Remote Procedure Call) to interact with the backend cluster. gRPC is an open source Remote Procedure Call (RPC) framework that can efficiently communicate between different computers and processes, and uses Protocol Buffers as its interface definition language, which can automatically generate client and server code, greatly simplifying the development of distributed applications. By using gRPC, CSI can conveniently communicate with the backend storage cluster, and at the same time, can provide a standard CSI interface for the Kubernetes cluster, achieving good scalability and maintainability.

[0038] Referring to FIG. 1, Figure 1 Figure 1 is a schematic diagram of the architecture of the provisioner component provided by the embodiment of the application. Wherein:

[0039] The ControllerServer 11 is an important component in the CSI provisioner 10 plugin, which is used to send a volume request to the ESSD cluster 13, and the volume request can include operations such as creation, deletion, expansion or query of the ESSD. The ControllerServer 11 is connected to the volume management service of the cluster through the gRPC protocol between the ControllerServer 11 and the ESSD cluster 13.​

[0040] Kubernetes 14, used to send instructions such as creating, deleting PVC, etc. to ControllerServer 11.

[0041] ControllerServer 11 is the core module inside the Provisioner component, which is used to receive relevant instructions sent from Kubernetes 14 and is responsible for converting them into operations on the backend storage.

[0042] In the embodiment, the ESSD cluster 13 includes a gRPC Server, which provides an interface for the gRPC server running on the remote ESSD cluster and providing volume management services externally. ControllerServer 11 also includes a gRPC client, which is a library located inside or tightly coupled with ControllerServer 11, and the client code is automatically generated by Protocol Buffers definition file.

[0043] In the embodiment, IdentityServer 12 is a standard component defined in the CSI specification, which is used to inform the caller of the name or version number of the CSI plugin. Kubernetes will first "identify" and "verify" this plugin by calling the GetPluginInfo and GetPluginCapabilities methods of IdentityServer 12, and then trust it and send more important CreateVolume or NodeStageVolume requests to it. It is the first step for CSI plugins and Kubernetes to establish trust and communication. Therefore, IdentityServer 12 can be responsible for identity-related requests sent by Kubernetes 14, such as obtaining CSI plugin information, etc.

[0044] (2) Node architecture design.

[0045] In the embodiment, the main component in Node is NodeServer, which is responsible for node-level operations such as mounting and unmounting volumes, and is mainly related to the host.

[0046] First, how to read and write high-performance cloud disks on physical machines needs to be solved. Currently, high-performance cloud disks such as ESSD basically use NVMe solid state disks as underlying storage media, and SPDK (Storage Performance Development Kit) has a natural advantage in the use scenario of NVMe+RDMA. The embodiments of the present application connect to a remote NVMe-oF target by installing the SPDKNVMe driver nvmf_tgt on the physical machine, expose block devices through RDMA, and interact with them in the same way as local NVMe SSDs, to achieve convenient access to high-performance cloud disks.

[0047] Moreover, the embodiments can make good use of the mechanisms such as large pages, polling, sub-kernels, and lock-free of the SPK high-performance kit, thereby reducing the system overhead caused by context switching, concurrent exclusion, and the like on the CPU (Central Processing Unit). In this way, the performance bottlenecks existing in the front-end network, the back-end network, and the disk writing can be solved, the response speed of I / O (Input / Output) can be improved, and the system performance can be improved.

[0048] Secondly, the present solution mounts the SPDK socket file “ / var / tmp / spdk.sock” into a pod (Kubernetes Pod, Kubernetes container group) to realize the communication and interaction between the Node plug-in and the nvmf_tgt service on the host machine. This is the default listening address used by the rpc client in SPDK when accessing, and developers or administrators can connect to this socket to perform various management and control operations on the SPDK application.

[0049] Finally, the embodiments realize specific operations on volumes through the “. / spdk / scripts / rpc.py” script. The rpc.py script file is an RPC client tool provided by SPDK, and the rpc functions provided by each module can be called through this script. Therefore, the embodiments only need to mount rpc.py and the directory where spdk.sock is located into the pod, and then interact with the nvmf_tgt on the host machine through the spdk-rpc command to realize operations such as creation, deletion, expansion, and Qos configuration of block devices. Moreover, the embodiments can modify the processing logic in the python file in the “. / spdk / python / spdk / rpc” directory to meet other development and debugging requirements, and have a certain extensibility.

[0050] This embodiment utilizes SPDK and NVMe-oF technologies to directly manage block devices on the host machine, fully leveraging the performance advantages of ESSD cloud disks and providing higher storage access performance. The embodiment also eliminates the need for integration and optimization of RDMA support in the source code, as this process is completed on the host machine and relies on the underlying network hardware's RDMA capabilities. Therefore, we only need to correctly configure and use these RDMA network devices. This embodiment simplifies the overall architecture's operation and management, offering a more direct and efficient approach than CephCSI.

[0051] Based on the above embodiments, in order to ensure data reliability, the embodiments also include a persistence design. The embodiments create an essd.volumes dictionary in memory to store ESSD cloud disk volume information, and also persist the volume information to a JSON (JavaScript Object Notation) file on the disk. Each time a volume is created or deleted, the relevant information is written to this JSON file so that the volume information can be recovered from the file when the plugin is restarted.

[0052] The in-memory `essd.volumes` dictionary provides fast access and update capabilities, while the JSON file on disk ensures the persistence and reliability of volume information. This design effectively avoids the loss of volume information due to plugin process restarts or node failures, thereby ensuring the consistency and availability of persistent volumes in the Kubernetes cluster.

[0053] like Figure 2 As shown, Figure 2 The JSON file writing process provided in this application embodiment can specifically include the following steps:

[0054] S21, Enter volume information.

[0055] In this embodiment, this step is the persistent data preparation phase. After the ControllerServer successfully executes an operation (such as creating or deleting a volume), it retrieves or generates detailed information (metadata) for that volume.

[0056] In this embodiment, volume information is typically organized within a Go language struct. The fields of this struct may include:

[0057] VolumeID: The unique identifier of the volume (the most critical field).

[0058] Capacity: The volume's capacity (e.g., how many GiB).

[0059] RequestName: Corresponding Kubernetes PVC name.

[0060] Status: Current state of the volume (e.g. Created, Deleting).

[0061] Creation time, cluster membership, etc.

[0062] Therefore, this step determines what data needs to be saved, which is the basis for all subsequent operations.

[0063] S22, convert the volume information structure to a json string through the json.Marshal interface.

[0064] In the embodiment, this step completes data serialization (Serialization), which converts the structure object in memory (Go's native data structure) into a platform-independent, language-independent string representation (JSON format).

[0065] Since serialization is the prerequisite for persistence, only by converting the data into a standard format (JSON) can it be reliably written to a file and correctly read in the future.

[0066] S23, generate the corresponding json file path according to the volume ID to ensure uniqueness.

[0067] In the embodiment, this step solves the organization and indexing problem of data storage, and can assign a unique storage location for the metadata of each volume. It can ensure that read and write operations on each volume do not interfere with each other, and when you need to find information about a certain volume, you can directly construct the file path through the ID to quickly locate it without the need to traverse all files, which can achieve fast retrieval. It can also prevent different volumes' information from being mistakenly overwritten.

[0068] S24, write the encoded JSON data to the file through the ioutil.WriteFile function.

[0069] In the embodiment, this step is the final execution step of data persistence (Persistence), which can permanently save the serialized JSON data from memory to the specified location on the disk. It is the core technical means to achieve high availability and state recovery capability. Only by writing data to non-volatile storage (disk) can the management state in memory (essd.volumes dictionary) be restored after the plugin restarts (json.Unmarshal), thereby avoiding information loss.

[0070] Embodiments can ensure that the CSI plugin itself manages the state will not be lost due to program restart, node restart and other failures. If the plugin Pod is scheduled to another node, the newly started instance can quickly take over the storage management task by reading these JSON files. And embodiments use the Go language standard library and the JSON general format to achieve a lightweight and highly reliable persistence solution without relying on external databases (such as MySQL, Redis), greatly simplifying the system architecture and deployment complexity.

[0071] As Figure 3 shown, Figure 3 The JSON file reading process schematic diagram provided by the embodiments of the present application can specifically include the following steps:

[0072] S31, input volume information.

[0073] In embodiments, this is the trigger and positioning starting point of data reading, which is different from S21. At this time, the purpose is not to input complete volume information, but to input the key identifier for finding the required volume information.

[0074] The "input" here is usually not manual input, but automatic call of program logic. For example:

[0075] Traverse recovery: when the component is started, in order to recover all states, it will try to read all JSON files under the persistent directory. At this time, the "input" can be regarded as the directory path (such as / var / lib / essd-csi / ).

[0076] On-demand query: when a request to process a specific volume is needed (such as deleting a volume with a known ID), the "input" at this time is the specific volume ID (VolumeID).

[0077] S32, generate the corresponding json file path according to the volume ID.

[0078] In embodiments, this step is to map the logical identifier to the physical storage location. It uses the same algorithm as the above step S23 when writing, which derives the exact location of the corresponding file on the disk from the volume ID. For example, string splicing is performed using the same rules as step S23. This design ensures the consistency of the read-write path, ensuring that the previously written file can be accurately found. Therefore, this step is the key to realizing reliable persistence. The unique and predictable file path is the guarantee for correct data retrieval.

[0079] S33, read the file content using ioutil, ReadFile function.

[0080] Embodiments load the non-volatile data stored on the disk into the memory by performing a physical read operation.

[0081] The `ioutil.ReadFile()` function is a convenient Go function for reading the entire contents of a file at once. It takes a file path as an argument and returns a []byte slice of the file contents and an error. If the file does not exist or the read fails, it returns the corresponding error, which the plugin can use to determine if the volume information may have been deleted or if error handling is required.

[0082] S34 is decoded into a recognizable structure using the json.Unmarsha function.

[0083] In this embodiment, this step completes data deserialization, which is the reverse process of step S22 above. It converts the generic, serialized JSON string back into a native data structure (struct) that can be directly understood and manipulated by the Go language program.

[0084] In this embodiment, after decoding, the volume information is no longer a meaningless string of characters, but a three-dimensional program object that restores all its attributes (such as capacity, status, name, etc.). This object is then populated into the essd.volumes dictionary, allowing the CSI plugin to "remember" the status of all managed volumes and continue to function normally and respond to requests.

[0085] Based on the above embodiments, in the embodiments provided in this application, SPDK can provide a high-performance, low-latency raw disk access method, avoiding the overhead of the traditional Linux block device stack. The embodiments can use the SPDK open source library to directly interact with the NVMe-oF protocol of the ESSD cloud disk, thus enabling direct management and mounting of the ESSD cloud disk as a raw disk storage, resulting in higher I / O performance, which is suitable for some application scenarios with high performance requirements.

[0086] Furthermore, since the design of the embodiment is closer to the access method of bare metal storage devices, it can create its own file system, which is compatible with most common file system types, giving the ESSDCSI plugin advantages in terms of performance and flexibility.

[0087] Specifically, such as Figure 4 As shown, Figure 4 A flowchart illustrating the process of mounting a file system on a nodeserver is provided. This process may include the following steps:

[0088] S41, acquire the mutex lock mutex.Lock0 to ensure thread safety.

[0089] Embodiments can prevent multiple mount / dismount requests from operating on the same volume or underlying resource at the same time, avoiding data competition and state confusion.

[0090] S42, parse request parameters: volume ID, volume name, volume size, volume type, target path, file system type.

[0091] Embodiments extract key information from requests sent from Kubernetes, which determines the specific object and target of all subsequent operations. For example: which volume, what file system, mount to where, etc.

[0092] S43, create a SPDKController corresponding to the target volume.

[0093] In embodiments, a logical controller (Controller) is created in the SPDK framework for the specified volume, which is the entry point for I / O interaction with NVMe devices.

[0094] S44, create an NVMf target for communication with SPDKController. Connect to the target and return the device path nvme_path after successful connection.

[0095] In embodiments, through the NVMe-oF protocol over RDMA network, connect to the remote ESSD cluster (Target), and generate a corresponding block device file (such as / dev / nvme1n1) in the local operating system. The nvme_path is the object of subsequent operations.

[0096] S45, os.Mkdir(mountPath, 0750) to create a mount target path.

[0097] In embodiments, the directory specified by the Pod can be created on the host's file system. For example, / var / lib / kubelet / ... / my-volume.

[0098] S46, mount the nvme_path device to the target path.

[0099] In embodiments, the local block device (nvme_path) obtained in the previous step can be mounted to the created directory (mountPath) by calling system functions (such as mount). At this point, the data channel has been completely connected.

[0100] Specifically, 1) Create a mount.SafeFormatAndMount instance, which encapsulates mount-related operations diskMounter:=&mount.SafeFormatAndMount{interface:mount.New(""),Exec:utilexec.NewO}.

[0101] 2) Create a file system using diskMounter.Exec.Command0: ext4: mkfs.ext4 -m0 -Enodiscard,lazy itable init=1,lazy journal init=1 $nvme path xfs: mkfs.xfs -K$nvme path.

[0102] 3) Mount the NVMe path to the target path opt := 0string{"netdev"} / / Inform the kernel that this is a file system that requires network support. This ensures that all network devices are ready when the system starts up before mounting the ESSD cloud disk, avoiding mounting failure due to network problems.

[0103] if volFsType == "xfs" { opt = append(opt, "nouuid")} / / XFS file system should not generate UUID (Universally Unique Identifier), which can avoid mount failure due to UUID changes. diskMounter.FormatAndMount(nvme path, mount path, file system type, opt).

[0104] S47 sets the mount point permissions to be readable, writable, and executable by everyone.

[0105] The implementation ensures that container processes within a Pod have sufficient permissions to read and write to mounted directories, thus guaranteeing storage availability.

[0106] S48, Update the volume information in the JSON file.

[0107] The implementation writes the successful result of this mount (e.g., updating the volume's mount status and mount point path) to a persistent JSON file. This ensures that even if the plugin restarts, it will still know which volume is mounted and where.

[0108] S49, release the mutex.

[0109] The embodiment releases the lock acquired at the beginning after the mounting operation is completed, allowing the system to process the next storage request for other volumes, thereby releasing resources and restoring concurrency.

[0110] Based on the above embodiment, the embodiment of the application provides a container-based cloud hard disk management method, as shown in the method can comprise the following steps: Figure 5

[0111] In step S510, the ControllerServer component of the CSI plugin communicates with the ESSD cluster based on the gRPC protocol, and manages the ESSD cloud hard disk in response to the persistent volume claim (PVC) request from the Kubernetes. The management can include life cycle management such as creation, deletion, or expansion.

[0112] In the embodiment, the ControllerServer component of the CSI plugin serves as a control center and communicates with the remote ESSD cluster management API through the efficient gRPC protocol. It is specially responsible for responding to the storage resource application (PVC) at the Kubernetes cluster level, and can realize the full life cycle management of the ESSD cloud hard disk, such as creation, deletion, or expansion. Further, it can ensure the on-demand allocation and elastic expansion of storage resources.

[0113] In step S520, the nvmf_tgt service of SPDK is run on the host machine, and interacts with the ESSD cloud hard disk in the ESSD cluster through the NVMe-oF protocol and the RDMA network, to create a block device (BDEV) corresponding to the ESSD cloud hard disk locally.

[0114] The embodiment establishes a high-performance data path, and on the computing node (host machine) that needs to access storage, a high-performance service nvmf_tgt of SPDK can be run to establish a low-latency and high-bandwidth network connection with the remote ESSD cluster by using the NVMe-oF over RDMA protocol. The core function of the service is to create a block device (BDEV) corresponding to the remote ESSD cloud hard disk in the local operating system, providing an object for subsequent mounting operations.

[0115] In step S530, the SPDK socket file spdk.sock and the directory of the SPDK remote procedure call script rpc.py on the host machine are mounted to the Pod running the NodeServer component of the CSI plugin.

[0116] ​In order to enable the NodeServer component running in the Pod to interact with the SPDK service (nvmf_tgt) on the host machine, the directory of the SPDK management socket (spdk.sock) and the management script (rpc.py) on the host machine are mounted into the Pod innovatively. This is equivalent to providing a "control channel" for the program inside the container, enabling it to send management instructions directly to the SPDK service on the host machine.

[0117] In step S540, upon receiving the Kubernetes request for mounting or unmounting the ESSD cloud disk, the rpc.py script is invoked inside the Pod through the NodeServer component, and based on the spdk-rpc command, the nvmf_tgt service on the host machine is interacted with to perform mounting or unmounting operations on the block device BDEV.

[0118] The embodiment describes the specific mounting / unmounting execution. When the NodeServer component receives the mounting / unmounting request from Kubernetes, it directly invokes the RPC script (rpc.py) of SPDK inside the Pod using the "control channel" provided in step S530, and interacts with the nvmf_tgt service on the host machine in the form of spdk-rpc command. Thus, mounting or unmounting operations on the local block device (BDEV) are realized, and the whole process of delivering storage resources to the application inside the Pod is finally completed.

[0119] The embodiment combines SPDK user space driver and NVMe-oF over RDMA technology to get rid of the overhead of traditional kernel storage stack, and at the same time solves the problem of interaction between CSI node components and host services by mounting sock files, finally realizing the containerized efficient use of high-performance cloud hard disks.

[0120] Based on the above embodiment, in another embodiment provided in the present application, when communicating with the ESSD cluster based on the gRPC protocol, the above step S510 can further include the following steps:

[0121] In step S511, the gRPC service interface is defined using Protocol Buffers.

[0122] In the embodiment, the communication protocol between the client (CSI Controller Server) and the server (ESSD cluster management API) is explicitly defined by using the Protocol Buffers (an interface definition language, IDL) developed by Google. This includes specifying the names of the remote procedure call (RPC) methods that can be called, the format of the request parameters and the format of the return response of the methods. This step ensures that both parties have a clear and consistent understanding of the data structure and service methods, and is the basis for subsequent automatic code generation.

[0123] In step S512, the client code is generated according to the gRPC service interface, and the volume management service API provided by the ESSD cluster is called through the generated client code.

[0124] The embodiment uses the compilation tools provided by the gRPC framework to automatically generate strongly typed client code (usually in Go, Java, etc.) from the.proto interface file defined in the previous step. The tedious network communication and serialization code can be manually written, greatly reducing the development workload and eliminating human errors. Subsequently, the Provisioner component directly uses these automatically generated, type-safe client code to call the specific volume management function APIs (such as CreateVolume, DeleteVolume) provided by the ESSD cluster.

[0125] In step S513, communication is performed with the ESSD cluster based on the volume management service API.

[0126] Based on the standardized client generated in the previous two steps and the defined interface, the CSI plugin and the ESSD cluster establish an efficient and reliable communication channel. All instructions and data for lifecycle management operations (creating, deleting, or expanding volumes) are transmitted in serialized Protocol Buffers binary format over the network through this channel, and finally interact with the backend storage system.

[0127] Therefore, the embodiment defines the interface, uses the IDL to design in advance, ensures the consistency of the contract, generates the code, uses the tool to automatically generate the code, guarantees the correctness and the development efficiency, and performs the calling, so that the remote calling is performed based on the generated code, and the function is implemented.

[0128] Based on the above embodiment, in another embodiment provided in the present application, the nvmf_tgt service of the SPDK running on the host machine interacts with the ESSD cloud hard disk in the ESSD cluster through the NVMe-oF protocol and the RDMA network, and the above step S520 can further include the following steps:

[0129] Step S521, load the NVMe-oF protocol and RDMA driver module, and start the nvmf_tgt service.

[0130] On the host machine, first load the necessary kernel or user space driver module, and prepare the software and hardware environment for using the NVMe-oF protocol and the RDMA network. Then, start the nvmf_tgt service of SPDK. The service is a high-performance user mode target program, which is used as an NVMe-oF initiator here, and is responsible for managing all block device operations through the NVMe-oF protocol. In this way, a high-speed data channel is prepared for subsequent establishment.

[0131] Step S522, discover the remote ESSD cloud disk through the RDMA network, and establish a connection with the ESSD cloud disk.

[0132] The embodiment establishes an end-to-end remote storage access channel, and the nvmf_tgt service actively discovers the NVMe-oF target exposed through the network in the ESSD cluster through the configured RDMA network (such as RoCEv2). After discovering the target, the initiator establishes a connection with the target and performs an NVMe-oF protocol handshake. Once the connection is successful, the remote ESSD cloud disk is presented as a standard block device (BDEV) (for example, / dev / nvme1n1) on the local host machine. This process realizes the localization abstraction of remote storage and provides an object for subsequent mounting operations.

[0133] The embodiment prepares the engine and establishes the channel, can configure the environment and start the high-performance user mode service (nvmf_tgt), discover and connect the remote storage resource through the RDMA network, and map it as a local device. Compared with the traditional use of kernel mode drivers (such as the NVMe-of TCP kernel driver) or general network protocols (such as iSCSI), the embodiment uses the advanced technology stack of SPDK and NVMe-oF over RDMA to construct the data channel, which can provide a technical basis for achieving the goal of ultra-low latency and high IOPS performance.

[0134] In the embodiment provided in the application, before running the nvmf_tgt service of SPDK on the host machine, the SPDK can also be configured to use large page memory and bind to a specific CPU core.

[0135] Specifically, in the embodiment, before starting the nvmf_tgt service of SPDK, the running environment of the nvmf_tgt service can be configured as follows:

[0136] Configure Hugepages: instructs SPDK to use a huge page memory pool instead of traditional 4KB memory pages. This greatly reduces the number of cache misses in the CPU TLB (Translation Lookaside Buffer), thus greatly reducing memory access latency, which is critical for handling high-frequency I / O requests.

[0137] Bind specific CPU cores (CPU Pining): bind the nvmf_tgt service process or its Polling thread to a specified one or more CPU physical cores. This avoids the cache invalidation and context switching overhead caused by process scheduling switching between different cores, and can ensure the stability and extremely low latency of the packet processing process.

[0138] Based on the above embodiments, in another embodiment provided in the application, the method can further include the following steps:

[0139] In step S550, after the creation or deletion operation of the ESSD cloud disk is performed, the volume information of the ESSD cloud disk is stored in a JSON file.

[0140] In the embodiment, whenever the ControllerServer successfully completes an operation that will change the management state of the cloud disk (such as creation or deletion), it will immediately write the latest metadata (such as volume ID, capacity, and state) of the volume involved in this operation in JSON format to a specified file on the host disk. The embodiment can ensure that any critical state change can be recorded in a timely and persistent manner on a non-volatile storage medium, and can ensure that the state change is safely saved, thereby avoiding the problem of memory data loss caused by accidental crashes of the plug-in process.

[0141] In step S560, when the Provisioner component or the Node component starts, load the volume information from the JSON file to restore the management state of the cloud disk by the CSI plug-in.

[0142] In the embodiment, when the Provisioner or Node component (usually due to its restart or scheduling migration in the form of a Pod) restarts, it will first read and parse all the previously persisted volume information from the JSON file on the disk before starting to process any new requests. Using this information, the component can completely rebuild its management state (for example, the essd.volumes dictionary) in memory, thereby "remembering" all existing volumes and their attributes. This enables the component to seamlessly take over subsequent management tasks and appears as a reliable service with state recovery capabilities to the outside. In this way, the continuity of the service can be guaranteed, and the component can quickly recover after restarting.

[0143] Therefore, the embodiment can realize the persistence and high availability of the CSI plugin management state without relying on any external database (such as ETCD, MySQL), only through the file system function of the operating system itself and the application layer coding (JSON serialization / deserialization), greatly simplifying the complexity and operation and maintenance cost of the system architecture, and ensuring the continuity of the business.

[0144] Based on the above embodiment, in another embodiment provided in the application, before the mounting or unmounting operation is performed on the block device BDEV, the method further includes the following steps:

[0145] In step S570, the parameter of the target file system type is received by the NodeServer component.

[0146] The embodiment provides an entry for personalized storage configuration. Before performing the mounting operation, the NodeServer component first parses and receives the target file system type (such as xfs, ext4) specified by the user from the request parameter passed by Kubernetes. The parameter is usually derived from the configuration of the PersistentVolumeClaim (PVC) and can reflect the characteristics of providing storage services flexibly according to application requirements.

[0147] The embodiment can reflect the flexibility of user configuration and meet the different performance and data management requirements of different applications for file systems.

[0148] In step S580, the NodeServer component calls the system command corresponding to the target file system type on the block device BDEV to autonomously create a file system. The file system includes xfs or ext4.

[0149] After obtaining the file system type parameter, the NodeServer component directly calls the corresponding system-level formatting command (such as mkfs.xfs or mkfs.ext4) on the locally mapped block device (BDEV) to create (format) the specified file system. This makes the embodiment not dependent on the file system pre-configured by the backend storage system or the host operating system, thereby providing great flexibility and wide compatibility. The embodiment can reflect the independence of the environment and ensure that the plugin can complete the final preparation of the storage medium regardless of the underlying environment.

[0150] The embodiment can build the required file system on the general block device as needed through standard system commands, and can completely decouple the dependence on specific backend storage or host environment.

[0151] In the case of dividing each functional module according to each function, the application provides a container-based cloud hard disk management device, which can be a server, a terminal or a chip applied to a server. Figure 6 An exemplary embodiment of the container-based cloud hard disk management device provided by the application is shown in the functional module schematic diagram. As shown in the figure, the container-based cloud hard disk management device comprises: Figure 6

[0152] The management module 61 is configured to communicate with the ESSD cluster based on the gRPC protocol through the ControllerServer component of the CSI plug-in, and perform life cycle management of creating, deleting or expanding the ESSD cloud hard disk in response to the persistent volume claim PVC request from the Kubernetes;

[0153] The interaction module 62 is configured to run the nvmf_tgt service of the SPDK on the host machine, interact with the ESSD cloud hard disk in the ESSD cluster through the NVMe-oF protocol and the RDMA network, and create a block device BDEV corresponding to the ESSD cloud hard disk locally;

[0154] The mounting module 63 is configured to mount the directory where the SPDK socket file spdk.sock and the SPDK remote procedure call script rpc.py are located on the host machine to the Pod running the NodeServer component of the CSI plug-in;

[0155] The processing module 64 is configured to, when receiving the mounting or unmounting request of the Kubernetes to the ESSD cloud hard disk, call the rpc.py script in the Pod through the NodeServer component, interact with the nvmf_tgt service of the host machine based on the spdk-rpc command, and perform mounting or unmounting operation on the block device BDEV.

[0156] In another embodiment provided by the application, the management module 61 is specifically configured to:

[0157] define the gRPC service interface using the Protocol Buffers;

[0158] generate the client code according to the gRPC service interface, and call the volume management service API provided by the ESSD cluster through the generated client code;

[0159] communicate with the ESSD cluster based on the volume management service API.

[0160] In another embodiment provided by the application, the interaction module 62 is specifically configured to:​

[0161] Load the NVMe-oF protocol and RDMA driver module, and start the nvmf_tgt service;

[0162] Discover the remote ESSD cloud disk through the RDMA network, and establish a connection with the ESSD cloud disk.

[0163] In another embodiment provided by the present application, the apparatus further comprises:

[0164] The configuration module is configured to configure the SPDK to use large page memory and bind to a specific CPU core.

[0165] In another embodiment provided by the present application, the apparatus further comprises:

[0166] The storage module is configured to, after the ESSD cloud disk is created or deleted, persistently store the volume information of the ESSD cloud disk in a JSON file;

[0167] The recovery module is configured to, when the Provisioner component or the Node component is started, load the volume information from the JSON file to restore the management state of the cloud disk by the CSI plug-in.

[0168] In another embodiment provided by the present application, the apparatus further comprises:

[0169] The parameter receiving module is configured to receive parameters of a target file system type through the NodeServer component;

[0170] The file system creating module is configured to, through the NodeServer component, call a system command corresponding to the target file system type on the block device BDEV to autonomously create a file system; wherein the file system includes an xfs or an ext4 type.

[0171] The electronic device provided by the embodiments of the present application comprises at least one processor, a memory for storing instructions executable by the at least one processor, and the at least one processor is configured to execute the instructions to implement the above-mentioned method disclosed by the embodiments of the present application.

[0172] Figure 7 The structure schematic diagram of the electronic device provided by an exemplary embodiment of the present application is shown in the figure. Figure 7 As shown in the figure, the electronic device 1800 comprises at least one processor 1801 and a memory 1802 coupled to the processor 1801, and the processor 1801 can execute the corresponding steps in the above-mentioned method disclosed by the embodiments of the present application.

[0173] The processor 1801 can also be referred to as a central processing unit (CPU), which can be an integrated circuit for processing digital signals. Each of the steps of the above-described methods disclosed in the embodiments of the present application can be completed by an integrated logic circuit of hardware or an instruction in the form of software in the processor 1801. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied in a hardware decoding processor for execution, or be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a memory 1802, such as a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, and other mature storage mediums in the art. The processor 1801 reads information in the memory 1802 and completes the steps of the above-described methods in combination with the hardware thereof.

[0174] In addition, various operations / processes according to the present application, when implemented by software and / or firmware, can be loaded from a storage medium or a network into a computer system having a dedicated hardware structure, such as the computer system 1900 shown below. Figure 8 The computer system 1900 shown below is installed with programs constituting the software, and when various programs are installed, the computer system can perform various functions, including functions such as those described above. Figure 8 A structural block diagram of a computer system according to an exemplary embodiment of the present application is provided.

[0175] The computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, tablets, personal digital assistants, servers, servers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the figures, their connections, and their functions, as well as their functions, are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0176] As shown in FIG. 19, the computer system 1900 includes one or more processors, such as a processor 1904. The processor 1904 is connected to a communication infrastructure 1902, such as a bus, a cross-over bar, or a network. Figure 8As shown, the computer system 1900 includes a computing unit 1901 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from the storage unit 1908 into a random access memory (RAM) 1903. Various programs and data required for the operation of the computer system 1900 can also be stored in the RAM 1903. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other through a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0177] Various components in the computer system 1900 are connected to the I / O interface 1905, including an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 can be any type of device that can input information to the computer system 1900, and can receive inputted digital or character information, as well as generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 1907 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0178] The computing unit 1901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1901 performs various methods and processes described above. For example, in some embodiments, the above-described methods disclosed by embodiments of the present application can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the above-described methods disclosed by embodiments of the present application by any other appropriate means (e.g., by means of firmware).

[0179] The embodiment of the present application further provides a computer readable storage medium, wherein when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method disclosed by the embodiment of the present application.

[0180] The computer readable storage medium in the embodiment of the present application can be a tangible medium, which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The computer readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specifically, the computer readable storage medium can include one or more wires, portable computer disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fibers, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0181] The computer readable medium can be included in the electronic device, or can exist separately from the electronic device.

[0182] The embodiment of the present application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method disclosed by the embodiment of the present application.

[0183] In the embodiments of the present application, the computer program code for performing the operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer.

[0184] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. The body of computer readable program code can be any suitable set of instructions stored on a non-transitory computer readable medium, such as a memory or hard disk, that, if executed by a processor, is able to operate as described herein. The computer readable medium can be portable, for example, a compact disc, a memory stick, or a hard disk. The computer readable medium can be non-portable, for example, a hard disk that is part of a computer.

[0185] The modules, units or components described in implementation of the present application can be implemented by software, or by hardware. In some cases, the name of the module, unit or component does not constitute a limitation on the module, unit or component itself.

[0186] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0187] The above description is merely exemplary of some embodiments of the present application and of the technology principle employed. It is understood that the scope of the disclosure of the present application is not limited to the specific combinations of technical features described above, but also covers other technical solutions formed by any combination of the technical features described above or their equivalent features, without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with technical features disclosed in the present application (but not limited to) having similar functions.

[0188] Although some specific embodiments of the present application have been described in detail by way of examples, it is to be understood that the examples are intended solely for the purposes of illustration and that the scope of the present application is not limited to the examples. Those skilled in the art shall understand that modifications can be made to the above embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A container-based cloud hard disk management method, characterized in that, The method comprises: A ControllerServer component of a CSI plugin communicates with an enhanced cloud hard disk (ESSD) cluster based on a Google remote procedure call (gRPC) protocol, and manages the ESSD cloud hard disk in response to a persistent volume claim (PVC) request from Kubernetes; An nvmf_tgt service of a storage performance development kit (SPDK) running on a host machine interacts with an ESSD cloud hard disk in the ESSD cluster through a network-based NVMe over Fabrics (NVMe-oF) protocol and a remote direct memory access (RDMA) network, to locally create a block device (BDEV) corresponding to the ESSD cloud hard disk; A SPDK socket file (spdk.sock) and a directory of a SPDK remote procedure call (rpc.py) script on the host machine are mounted to a Pod running a NodeServer component of the CSI plugin; Upon receiving a mounting or unmounting request for the ESSD cloud hard disk from Kubernetes, the rpc.py script is invoked in the Pod through the NodeServer component, and a spdk-rpc command is used to interact with the nvmf_tgt service of the host machine to perform mounting or unmounting operations on the BDEV.

2. The method of claim 1, wherein, Communicating with an ESSD cluster based on a gRPC protocol comprises: Defining a gRPC service interface using Protocol Buffers; Generating client code according to the gRPC service interface, and invoking a volume management service API provided by the ESSD cluster through the generated client code; Communicating with the ESSD cluster based on the volume management service API.

3. The method of claim 1, wherein, The nvmf_tgt service of the SPDK running on the host machine interacts with the ESSD cloud hard disk in the ESSD cluster through the NVMe-oF protocol and the RDMA network, comprising: Loading an NVMe-oF protocol and an RDMA driver module, and starting the nvmf_tgt service; Discovering a remote ESSD cloud hard disk through the RDMA network, and establishing a connection with the ESSD cloud hard disk.

4. The method of claim 1, wherein, Before the nvmf_tgt service of the SPDK running on the host machine, the method further comprises: Configuring the SPDK to use large page memory and bind to specific central processing unit (CPU) cores.

5. The method of claim 1, wherein, The method further comprises: After the creation or deletion operation on the ESSD cloud hard disk, persistently storing volume information of the ESSD cloud hard disk in a JSON file; When a Provisioner component or a Node component starts, loading the volume information from the JSON file to restore the management state of the CSI plugin for the cloud hard disk.

6. The method of claim 1, wherein, Before the mounting or unmounting operation on the BDEV, the method further comprises: Receiving a parameter of a target file system type through the NodeServer component; The NodeServer component calls a system command corresponding to the target file system type on the block device BDEV to autonomously create a file system.

7. The method of claim 6, wherein, The file system includes an xfs or ext4 type.

8. The method of claim 1, wherein, The management of the ESSD cloud disk includes: The life cycle management of the ESSD cloud disk includes creation, deletion, or expansion. 9.A container-based cloud hard disk management apparatus, characterized in that, The device includes: A management module configured to communicate with an ESSD cluster based on a gRPC protocol through a ControllerServer component of a CSI plug-in, and perform ESSD cloud disk management in response to a persistent volume claim PVC request from Kubernetes; An interaction module configured to run an nvmf_tgt service of SPDK on a host machine, and interact with an ESSD cloud disk in the ESSD cluster through an NVMe-oF protocol and an RDMA network to locally create a block device BDEV corresponding to the ESSD cloud disk; A mounting module configured to mount a directory where a SPDK socket file spdk.sock and a SPDK remote procedure call script rpc.py are located to a Pod running a NodeServer component of the CSI plug-in; A processing module configured to, when receiving a mounting or unmounting request of the ESSD cloud disk from Kubernetes, call the rpc.py script in the Pod through the NodeServer component, and interact with the nvmf_tgt service of the host machine based on a spdk-rpc command to perform mounting or unmounting operations on the block device BDEV.

10. An electronic device, comprising: Comprise: At least one processor; Memory for storing instructions executable by the at least one processor; Wherein the at least one processor is configured to execute the instructions to implement the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can perform the method of any one of claims 1-8.

12. A computer program product, characterised in that, Comprise a computer program, which is executed by a processor to implement the method of any one of claims 1-8.

Images