Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: acquiring target authentication information of the distributed storage system; generating a target file comprising distributed object storage block equipment information; using the target authentication information to execute a deployment operation of the containerized data import related component, and after the deployment operation is completed, adding a preset label in a persistent volume statement corresponding to the data storage volume; obtaining target encrypted information required for mapping of the distributed object storage block device from the data storage volume, and mounting the target encrypted information to a preset directory; and adding the target authentication information to configuration information of the target virtual machine, and starting the target virtual machine. The technical problem of low data transmission efficiency caused by the fact that the target virtual machine management platform cannot directly access the target distributed storage system is solved.

Description

Technical Field

[0001] The present application relates to the fields of IT and software development, and specifically, to a data processing method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] In the current cloud-native environment, Kubernetes is the de facto standard for container orchestration, and its richness and flexibility provide strong support for cloud services. As a project that runs and manages virtual machines on Kubernetes, the virtual machine management platform KubeVirt makes up for the shortcomings of Kubernetes in handling traditional virtualization workloads. However, in terms of data access and storage management of virtual machines, KubeVirt relies on traditional storage solutions, which may become a bottleneck in high-performance and high-throughput scenarios.

[0003] In KubeVirt's storage solution, the Containerized Data Importer (CDI) plays a key role. CDI is responsible for creating and managing data volumes for use by virtual machines. When a user requests to create a Block-type data storage volume (DataVolume, DV) based on Ceph's reliable autonomous distributed object storage block device (RADOS Block Device, RBD), CDI will monitor its status. Once the DataVolume is ready, the data will be imported into a Persistent Volume Claim (PVC), which will be mounted to the specified path in the container by the host's kernel layer through the Ceph Container Storage Interface (CSI) driver. Subsequently, when KubeVirt uses the QEMU process to start the virtual machine, it will indirectly access the data blocks stored in the PVC from the path in the container.

[0004] However, this traditional access path has some key problems. First, the data access path is long, involving virtual machines, QEMU, path mapping in containers, host kernels, and finally Ceph RBD. Each step may introduce additional latency and performance overhead. Second, due to differences in host kernel versions, different kernels may not be compatible with specific versions of the librbd library, which limits the stability and performance of virtual machines' access to Ceph storage. In addition, data read and write operations in this mode need to go through multiple layers of conversion, which not only increases computational complexity, but may also lead to waste of data duplication and reduce the efficiency of storage resource utilization.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The present application provides a data processing method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem of low data transmission efficiency caused by the inability of a target virtual machine management platform to directly access a target distributed storage system.

[0007] According to one aspect of the present application, a data processing method is provided, including: obtaining target authentication information of a distributed storage system, and storing the target authentication information in a container management system; when the container management system creates a block-type storage volume, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset tag, generating a target file including distributed object storage block device information; using the target authentication information to perform a deployment operation of a containerized data import-related component, after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, adding a preset tag to the persistent volume declaration corresponding to the data storage volume; in the process of starting a target container set by a virtual machine instance in a virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset tag, obtaining target encryption information required for mapping the distributed object storage block device from the data storage volume, and mounting the target encryption information to a preset directory; if it is detected that the target authentication information exists in the preset directory, adding the target authentication information to the configuration information of the target virtual machine, and starting the target virtual machine.

[0008] Optionally, if it is detected that the target authentication information exists in the preset directory, the method further includes: generating an encrypted file corresponding to the target authentication information, and calling a preset interface to access the storage part in the distributed storage system.

[0009] Optionally, starting the target virtual machine includes: traversing the disk to be added to the target virtual machine; when traversing the disk to be added to the target virtual machine, if the block to be mounted is a target file and the distributed object storage block device information in the target file is readable information, adding monitor information to the distributed object storage block device information, and adding disk information to the target virtual machine in the form of a network disk; after adding the disk information to the target virtual machine, starting the target virtual machine.

[0010] Optionally, storing the target authentication information in the container management system includes: storing the target authentication information in the form of a first encrypted object in the namespace of the containerized data import-related component of the virtual machine management platform in the container management system, and adding the target authentication information to the configuration information of the containerized data import-related component.

[0011] Optionally, after adding a preset tag to the persistent volume declaration corresponding to the data storage volume, the method also includes: reading target authentication information and storing the target authentication information in the form of a second encrypted object; and accessing the storage part in the distributed storage system through the target authentication information in the second encrypted object.

[0012] Optionally, the preset tag includes: "cdi.kubevirt.io / storage.directio" = true.

[0013] Optionally, mounting the target encrypted information into a preset directory includes: mounting the target encrypted information into the preset directory in a read-only form, wherein the preset directory includes: / etc / ceph.

[0014] According to another aspect of the present application, a data processing device is also provided, including: an acquisition module, used to acquire target authentication information of a distributed storage system and store the target authentication information in a container management system; a generation module, used to generate a target file including distributed object storage block device information when creating a block-type storage volume in the container management system, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset tag; an adding module, used to use the target authentication information to perform a deployment operation of containerized data import-related components, and after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, add the preset tag to the persistent volume declaration corresponding to the data storage volume; a processing module, used to obtain the target encryption information required for mapping the distributed object storage block device from the data storage volume during the process of starting the target container set of the virtual machine instance, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset tag, and mount the target encryption information to a preset directory; a startup module, used to add the target authentication information to the configuration information of the target virtual machine if it is detected that the preset directory exists. And start the target virtual machine.

[0015] According to another aspect of the present application, a non-volatile storage medium is provided, the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above data processing method.

[0016] According to another aspect of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above data processing method is executed when the program is run.

[0017] According to yet another aspect of the present application, a computer program is provided, wherein the above data processing method is implemented when the computer program is executed by a processor.

[0018] According to another aspect of the present application, a computer program product is provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above data processing method is implemented.

[0019] In the present application, the target authentication information of the distributed storage system is obtained and stored in the container management system; when the container management system creates a block type storage volume, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset tag, a target file including distributed object storage block device information is generated; the target authentication information is used to perform the deployment operation of the containerized data import related components. After the deployment operation is completed, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, the preset tag is added to the persistent volume declaration corresponding to the data storage volume; in the process of starting the target container set by the virtual machine instance in the virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset tag, the target encryption information required for mapping the distributed object storage block device is obtained from the data storage volume, and the target encryption information is mounted to the preset directory; if it is detected that the target authentication information exists in the preset directory, the target authentication information is added to the configuration information of the target virtual machine, and the target virtual machine is started, so as to achieve the purpose of the target virtual machine management platform directly accessing the target distributed storage system, thereby achieving the technical effect of improving the data transmission efficiency, and thus solving the technical problem of low data transmission efficiency caused by the inability of the target virtual machine management platform to directly access the target distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of an optimization target of a data processing method according to an embodiment of the present application;

[0023] Figure 3 is a flow chart of a method for starting a virtual machine according to an embodiment of the present application;

[0024] Figure 4 is a structural diagram of a data processing device according to an embodiment of the present application;

[0025] Figure 5It is a hardware structure block diagram of a computer terminal according to a data processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0030] Step S101, obtaining target authentication information of a distributed storage system, and storing the target authentication information in a container management system.

[0031] Among them, distributed storage systems such as Ceph provide high-availability object storage, block storage and file system storage services, aiming to build large-scale, high-performance, reliable and scalable storage solutions. The design goal of Ceph is to achieve data center-level storage resource pooling while ensuring data security and consistency. It uses an algorithm called CRUSH (Controlled Replication Under Scalable Hashing) to distribute data and replicas, ensuring balanced distribution of data among multiple nodes in the storage cluster, and can tolerate node failures without data loss. Container management systems such as Kubernetes (abbreviated as "k8s") is an open source container orchestration platform for automating the deployment, expansion and management of containerized applications. Kubernetes provides an efficient and scalable way to run containerized applications. It decomposes applications into smaller, manageable services or microservices and packages these services in containers. With Kubernetes, users can define the resource requirements of the container cluster, the network connection method of the service, the storage requirements, and how to automatically expand the application to adapt to different workloads. Kubernetes' automatic scheduling mechanism can ensure that the application container runs on the best node in the cluster, while providing health checks, self-healing and load balancing functions.

[0032] In step S101, create a Kubernetes Secret to store Ceph authentication information, where Secret is an object type in Kubernetes that is used to store sensitive information such as passwords, SSH keys, TLS certificates, or database credentials. The Kubernetes Secret includes the username, key, and other necessary information required to access Ceph storage. The Secret can be named "userName-libvirt" and stored in the "kubevirt-cdi" namespace of Kubernetes. Add this Secret to the CDI Config through the Kubernetes API to ensure that the containerized data importer (CDI) can access the authentication information.

[0033] Step S102, when the container management system creates a block-type storage volume, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset label, a target file including distributed object storage block device information is generated.

[0034] In step S102, when creating a block-type DataVolume in Kubernetes, CephCSIDriver checks the label "cdi.kubevirt.io / storage.directio" on the PVC. If the label is set to true, CSIDriver will not perform the traditional RBD mapping operation, but generate a file containing the RBD name and related RBD information. This file will contain all the RBD mapping information required to start the virtual machine without conversion through the host kernel.

[0035] Step S103, use the target authentication information to execute the deployment operation of the containerized data import related components. After completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the protocol description information corresponding to the data storage volume is true, add a preset tag to the persistent volume declaration corresponding to the data storage volume.

[0036] In step S103, during the CDI deployment process, the CDI component is started using the authentication information stored in step S101. Once CDI has completed the preparation and storage of data in the DataVolume, it will check the direct IO attribute (directio) in the spec description of the DataVolume. If this attribute is set to true, CDI will add a preset tag "cdi.kubevirt.io / storage.directio" = true to the corresponding PVC. This indicates that the data on this PVC can be directly accessed by the virtual machine without the traditional kernel layer mapping.

[0037] Step S104, during the process of starting the target container set by the virtual machine instance in the virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset label, the target encryption information required for mapping the distributed object storage block device is obtained from the data storage volume, and the target encryption information is mounted to the preset directory.

[0038] Among them, virtual machine management platforms such as KubeVirt, KubeVirt is an open source project that provides a set of extensions for Kubernetes that enable Kubernetes to manage and run virtual machines, just like managing containers. The goal of KubeVirt is to combine virtualization technology with containerization technology in order to run and manage virtualized workloads in Kubernetes clusters. With KubeVirt, users can use Kubernetes' APIs and CLI tools (such as kubectl) to operate virtual machines, including creating, starting, stopping, and deleting virtual machines. KubeVirt also provides advanced features such as virtual machine lifecycle management, virtual machine image management, virtual machine network configuration, and storage volume management.

[0039] In step S104, when KubeVirt starts a virtual machine instance (VMI), it checks whether the preset tag "cdi.kubevirt.io / storage.directio" = true exists on the PVC corresponding to the DataVolume. Once this tag is detected, the virt-launcher component will read the complete RBD mapping encryption information required from the DataVolume and mount it in read-only form to the preset directory " / etc / ceph" for use by the virtual machine instance.

[0040] It should be noted that the virt-launcher component is a key component in the KubeVirt project. It is responsible for starting and managing virtual machine instances (Virtual Machine Instance, VMI) on the nodes of the Kubernetes cluster. Specifically, the main responsibilities of virt-launcher include: 1. Lifecycle management of virtual machine instances: virt-launcher monitors events and state changes related to virtual machine instances. When the KubeVirt API Server receives a request to create a VMI, it notifies virt-launcher to start the virtual machine on the appropriate node. virt-launcher also handles operations such as stopping, restarting, and deleting virtual machines. 2. Preparation before virtual machine startup: Before starting a virtual machine, virt-launcher will perform a series of preparation operations, such as setting the network configuration, disk mapping, memory and CPU resources of the virtual machine. It also handles the storage volumes that the virtual machine depends on, such as loading data from storage systems such as Ceph RBD, NFS, iSCSI, and mapping it to the virtual machine. 3. QEMU process management: virt-launcher will start the QEMU process to perform hardware simulation and operation of the virtual machine. It passes the virtual machine's configuration information to QEMU, including disk mapping, network interface, CPU, memory and other parameters. Virt-launcher also monitors the status of the QEMU process to ensure the normal operation of the virtual machine.

[0041] QEMU (Quick Emulator) is an open source machine simulator and virtualization technology that can create and run virtual machines. QEMU supports multiple processor architectures and operating systems, and can be used as a complete system simulator or only simulate partial behavior of specific hardware devices. In virtualization scenarios, QEMU acts as a software-level CPU and hardware device simulator, allowing multiple independent virtual machines to run on a physical host. The QEMU process refers to a program instance that runs the QEMU software to simulate the hardware environment. When KubeVirt creates a virtual machine instance, it starts the QEMU process to perform hardware simulation, which includes: 1. Simulating the CPU and memory of the virtual machine, allowing the virtual machine to run different operating systems and applications. 2. Simulating various storage devices, such as hard disks, optical drives, etc., and supporting multiple storage backends, including local files, network storage (such as NFS, iSCSI, Ceph RBD), etc. 3. Simulating virtual network adapters, allowing virtual machines to communicate with external networks through virtual network interfaces. 4. For virtual machines that require a graphical user interface, QEMU can simulate graphics adapters and displays, and support remote display protocols such as Spice and VNC. There is no direct physical connection between the QEMU process and the actual operating system and hardware. Instead, the virtual machine can run on the physical host through software simulation. In KubeVirt, the QEMU process can be started and managed through the libvirt interface. Libvirt is a common API for managing and accessing virtualized resources. It provides support for multiple virtualization technologies, including QEMU / KVM.

[0042] Step S105: If it is detected that the target authentication information exists in the preset directory, the target authentication information is added to the configuration information of the target virtual machine, and the target virtual machine is started.

[0043] In step S105, before the virtual machine is started, virt-launcher will check whether there is authentication information in the " / etc / ceph" directory. If it exists, it will generate the corresponding secret.xml file and define these secrets through the virsh interface secret-define so that the virtual machine can access Ceph storage. When defining the virtual machine, the authentication information and RBD information are integrated into the virtual machine configuration. When virt-launcher traverses the disk information that needs to be added to the virtual machine, if it is found that the disk to be mounted is block type RBD information, the disk information is added to the virtual machine configuration in the form of networkdisk. Finally, QEMU is called through libvirt to start the virtual machine, so that the virtual machine can directly access Ceph RBD and achieve efficient data reading and writing.

[0044] According to the above steps, the present application has the following advantages: 1. Reduce the access path: the traditional path is virtual machine->QEMU->container path mapping>host kernel->Ceph RBD, while the optimized path is virtual machine->QEMU->CephRBD, which eliminates the path mapping in the container and the host kernel, thereby reducing latency and CPU usage. 2. Specify librbd access library: By directly calling a specific version of the librbd library through QEMU, the interaction stability and performance with Ceph RBD can be ensured, avoiding the problem of librbd library version differences caused by different host kernel versions. 3. Improve compatibility and stability: Since QEMU communicates directly with Ceph RBD and is not affected by the host kernel version, the compatibility and stability of the entire system can be improved. 4. Optimize read and write performance: By reducing access paths and specifying librbd library versions, read and write performance can be further optimized, data transmission delays can be reduced, and the response speed and overall performance of the virtual machine can be improved.

[0045] The following Figure 1 The steps shown are exemplary and explanatory.

[0046] According to some optional embodiments of the present application, if target authentication information is detected in a preset directory, the following steps may also be performed: generating an encrypted file corresponding to the target authentication information, and calling a preset interface to access the storage part in the distributed storage system.

[0047] In the above embodiment, in the KubeVirt environment, if it is detected that the target authentication information (Ceph's authentication information) exists in a preset directory (such as / etc / ceph), the following steps can be performed to securely access the storage part in the distributed storage system (such as Ceph): Use the tools or libraries provided by Ceph (such as ceph-authtool or librbd library) to generate an encrypted file containing the authentication information. Use the interface of libvirt or QEMU to process the key ring and access storage. For CephRBD, the virt-launcher component calls the secret-define interface of libvirt to define a secret, which is used to store Ceph's authentication information. Subsequently, this secret is referenced in the virtual machine definition to ensure that the virtual machine can access Ceph storage in a secure manner.

[0048] According to some other optional embodiments of the present application, starting the target virtual machine can be achieved by the following method: traversing the disk to be added to the target virtual machine; when traversing the disk to be added to the target virtual machine, if the block to be mounted is a target file and the distributed object storage block device information in the target file is readable information, adding monitor information to the distributed object storage block device information, and adding disk information to the target virtual machine in the form of a network disk; after adding the disk information to the target virtual machine, starting the target virtual machine.

[0049] In the above embodiment, in the KubeVirt environment, traversing the disks to be added to the target virtual machine and determining how to mount them according to the disk type and storage information are key steps to enable the virtual machine to efficiently access Ceph storage. The following is a detailed explanation of this process: When virt-launcher is ready to start a target virtual machine, it traverses all disk information contained in the definition of the target virtual machine. The disk information contains configuration parameters such as the disk type, size, and storage source. During the traversal process, virt-launcher checks the storage source of each disk. If the disk source is a target file and the file contains readable distributed object storage block device information, i.e., RBD information, then virt-launcher will further process the disk. For files marked as directly accessible to RBD, virt-launcher reads the RBD information from the file, which includes at least the pool name, image name, and authentication information of the RBD. Then, virt-launcher adds monitor (mon) information to the RBD information. The monitor is the component in the Ceph cluster responsible for managing the metadata of the block device image. The addition of mon information is to ensure that QEMU can directly find and access the correct Ceph cluster. Next, virt-launcher will use the configuration containing the RBD and mon information to add disk information to the target virtual machine in the form of a network disk. This means that the virtual machine's disk will be mounted directly in the form of Ceph RBD, rather than the traditional mapping through the file system or host kernel. After all the disk information has been properly added to the virtual machine configuration, virt-launcher will pass the configuration information to QEMU and start the target virtual machine. The virtual machine can now directly access the data in Ceph RBD without going through an additional intermediate layer.

[0050] In some optional embodiments of the present application, storing the target authentication information in the container management system can be achieved by the following method: storing the target authentication information in the form of a first encrypted object in the namespace of the containerized data of the virtual machine management platform in the container management system and importing the target authentication information into the namespace of the relevant component, and adding the target authentication information to the configuration information of the containerized data import-related component.

[0051] In the above embodiment, first, the target authentication information (such as Ceph's authentication key and ID) is converted into an encrypted Secret object. In Kubernetes, Secret is a data type used to store sensitive data, such as passwords, keys, etc., which are stored in the cluster and can be mounted in an encrypted form in a Pod. By creating a Secret object, the authentication information can be securely stored in the Kubernetes cluster instead of hard-coded in the configuration file or source code. The created Secret object will be stored in the namespace of the KubeVirt CDI-related component, such as the kubevirt-cdi namespace. Namespaces are used in Kubernetes to logically isolate resources, which helps to organize and manage resources and improve security. Subsequently, this Secret object containing the authentication information is added to the configuration of CDI-related components (such as cdi-deployment). In this way, when the component runs, it will have access to Ceph storage. The CDI component can read the authentication information in the Secret for subsequent storage operations, such as importing virtual machine images to Ceph RBD.

[0052] As some optional embodiments of the present application, after adding a preset tag in the persistent volume declaration corresponding to the data storage volume, the following steps can also be performed: read the target authentication information and store the target authentication information in the form of a second encrypted object; access the storage part in the distributed storage system through the target authentication information in the second encrypted object.

[0053] Understandably, to further increase security, the authentication information should be encrypted even when stored. Therefore, the authentication information is read and encrypted into a second encrypted object. In a Kubernetes environment, this means creating a new Secret object to ensure that the information is encrypted even when stored and cannot be accessed by unauthorized users or services.

[0054] Figure 2 is a schematic diagram of an optimization target of a data processing method according to an embodiment of the present application, namely, Figure 1 A schematic diagram of the optimization objective of the method shown is Figure 2As shown in the figure, the client sends a virtual machine management request to KubeVirt (WM) through the Kubernetes API Server. KubeVirt manages the life cycle of the virtual machine through the virt-launcher component and the libvirt interface. The virt-launcher detects the storage information that the virtual machine needs to access, and directly calls the librbd library to interact with the Ceph cluster instead of performing RBD mapping through the host kernel, which reduces the level of the data access path. The virtual machine manager Hypervisor directly uses the RBD information provided by librbd to start the virtual machine, allowing the virtual machine to directly access the data in the Ceph storage and improve I / O efficiency. The Ceph cluster provides data storage services, and the virtual machine communicates directly with it through the improved path.

[0055] Among them, the client can be any entity that requests KubeVirt services, such as end users, automated scripts, or applications. The client interacts with KubeVirt through the Kubernetes API Server to request the creation, start, stop, or deletion of virtual machine instances (VMIs). The virtual machine VM is the main resource provided by KubeVirt. The virtual machine is created by the client through the Kubernetes API. It can run on nodes managed by KubeVirt and access block devices in the Ceph storage system. In the KubeVirt architecture, the virtual machine manager Hypervisor can be QEMU or KVM, which is a virtualization layer that runs directly on the physical host and is responsible for managing the operating environment of the virtual machine, including virtualized CPU, memory, storage, network, etc. librbd is a library used to manage block devices (RBD) in the Ceph storage system. In Figure 2 In the improved architecture, librbd is directly called by QEMU to bypass the traditional host kernel path and realize direct communication between virtual machines and Ceph storage. Ceph cluster CCluster is a distributed storage system that can provide block storage, object storage, and file system storage services. In the KubeVirt environment, Ceph block storage (RBD) is used as the backend storage for virtual machine disks.

[0056] like Figure 1 and Figure 2 As shown, this application directly connects to Ceph RBD through the QEMU process inside KubeVirt, which can effectively reduce data access paths and improve data transmission efficiency. This method avoids the extra overhead caused by RBD mapping through the host kernel, because the data no longer needs to pass through the host kernel layer, but is directly passed from the virtual machine to QEMU, and then QEMU directly interacts with Ceph RBD.

[0057] This application provides a detailed solution that enables kubevirt virtual machines to use ceph directly, which is very different from the kernel mapping form. In addition, this application also has the following advantages: 1. Through the direct IO mechanism, the virtual machine can directly access the Ceph block device, avoiding the overhead of the traditional file system layer, thereby significantly improving the data read and write speed and I / O performance. 2. Reduce the intermediate links. The traditional RBD mapping method requires multiple levels of conversion to allow the virtual machine to access the storage, and this patented technology reduces these intermediate links, making the data path shorter and reducing latency. 4. Through dynamic detection and application-specific tags, the system can automatically select the optimal data access mode according to different workload requirements, enhancing the flexibility and adaptability of the system, and can freely choose whether to connect directly to ceph. 5. Since the traditional kernel mounting depends on kernel compatibility, after QEMU is directly connected, only the compatibility of the library brought by the application needs to be considered, which enhances compatibility. 6. Due to the reduction of unnecessary data copying and conversion, the system can use storage resources more efficiently, reduce the total cost of ownership, and improve resource utilization.

[0058] Figure 3 is a flow chart of a method for starting a virtual machine according to an embodiment of the present application, such as Figure 3 As shown, the method comprises the following steps:

[0059] Step S301, virt-lancher is started, that is, virt-launcher is started and is ready to initialize and configure the virtual machine instance.

[0060] Step S302: Check whether / etc / ceph exists. virt-launcher checks whether the preset directory / etc / ceph exists. This directory is used to store authentication information and configuration for accessing the Ceph storage system. If it exists, the process continues to step S303.

[0061] Step S303, if the / etc / ceph directory exists, virt-launcher will use the secret-define command of the libvirt tool virsh to define a secret, which contains the authentication information of Ceph. This step is to ensure that the virtual machine can securely access Ceph storage instead of directly exposing sensitive information in the Pod.

[0062] Step S304, after step S303, or when the etc / ceph directory does not exist, traverse the disk information. Specifically, virt-launcher starts to traverse all disk information of the target virtual machine, and checks the source and type of each disk.

[0063] Step S305, in the process of traversing the disk information, determine whether each disk is an RBD type file block, that is, whether it comes from Ceph RBD. If so, the process goes to step S306. If not, continue to traverse other disk information.

[0064] Step S306, for the disk marked as RBD type, virt-launcher will mount the disk in the form of networkdisk, which means that the virtual machine can directly communicate with Ceph RBD and access storage resources without mapping through the host kernel.

[0065] Step S307, after completing the disk information processing and mounting, virt-launcher will continue to execute the virtual machine startup process. The virtual machine can now access the storage resources provided by Ceph RBD and perform data interaction in a direct IO manner.

[0066] Figure 4 is a structural diagram of a data processing device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0067] The acquisition module 41 is used to acquire target authentication information of the distributed storage system and store the target authentication information in the container management system.

[0068] The generating module 42 is used to generate a target file including the distributed object storage block device information when the container management system creates a block type storage volume and detects that the persistent volume declaration corresponding to the storage volume includes a preset label.

[0069] Add module 43, which is used to use the target authentication information to execute the deployment operation of the containerized data import related components. After completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, add a preset label to the persistent volume declaration corresponding to the data storage volume.

[0070] Processing module 44 is used to obtain target encryption information required for mapping the distributed object storage block device from the data storage volume if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset label during the process of starting the target container set of the virtual machine instance, and mount the target encryption information to a preset directory.

[0071] The start module 45 is used to add the target authentication information to the configuration information of the target virtual machine and start the target virtual machine if it is detected that the target authentication information exists in the preset directory.

[0072] Optionally, if it is detected that the target authentication information exists in the preset directory, the method further includes: generating an encrypted file corresponding to the target authentication information, and calling a preset interface to access the storage part in the distributed storage system.

[0073] Optionally, starting the target virtual machine includes: traversing the disk to be added to the target virtual machine; when traversing the disk to be added to the target virtual machine, if the block to be mounted is a target file and the distributed object storage block device information in the target file is readable information, adding monitor information to the distributed object storage block device information, and adding disk information to the target virtual machine in the form of a network disk; after adding the disk information to the target virtual machine, starting the target virtual machine.

[0074] Optionally, storing the target authentication information in the container management system includes: storing the target authentication information in the form of a first encrypted object in the namespace of the containerized data import-related component of the virtual machine management platform in the container management system, and adding the target authentication information to the configuration information of the containerized data import-related component.

[0075] Optionally, after adding a preset tag to the persistent volume declaration corresponding to the data storage volume, the method also includes: reading target authentication information and storing the target authentication information in the form of a second encrypted object; and accessing the storage part in the distributed storage system through the target authentication information in the second encrypted object.

[0076] Optionally, the preset tag includes: "cdi.kubevirt.io / storage.directio" = true.

[0077] Optionally, mounting the target encrypted information into a preset directory includes: mounting the target encrypted information into the preset directory in a read-only form, wherein the preset directory includes: / etc / ceph.

[0078] It should be noted that the above Figure 4 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0079] It should be noted that Figure 4 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0080] Figure 5 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 5As shown, the computer terminal 50 may include one or more (502a, 502b, ..., 502n are used to illustrate) processors 502 (the processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission module 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown.

[0081] It should be noted that the one or more processors 502 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 50. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0082] The memory 504 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, realizing the above-mentioned data processing method. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 504 may further include a memory remotely arranged relative to the processor 502, and these remote memories may be connected to the computer terminal 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The transmission module 506 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 50. In one example, the transmission module 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0084] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 50 .

[0085] It should be noted that, in some optional embodiments, the above Figure 5 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 5 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0086] It should be noted that Figure 5 The computer terminal shown is used to execute Figure 1 The data processing method shown, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0087] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above data processing method.

[0088] A program for a non-volatile storage medium to perform the following functions: obtaining target authentication information of a distributed storage system and storing the target authentication information in a container management system; when the container management system creates a block-type storage volume, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset tag, generating a target file including distributed object storage block device information; using the target authentication information to perform deployment operations of components related to containerized data import, after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, adding a preset tag to the persistent volume declaration corresponding to the data storage volume; in the process of starting a target container set by a virtual machine instance in a virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset tag, obtaining target encryption information required for mapping the distributed object storage block device from the data storage volume, and mounting the target encryption information to a preset directory; if it is detected that the target authentication information exists in the preset directory, adding the target authentication information to the configuration information of the target virtual machine, and starting the target virtual machine.

[0089] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above data processing method is executed when the program is run.

[0090] The processor is used to run a program that performs the following functions: obtaining target authentication information of a distributed storage system and storing the target authentication information in a container management system; when the container management system creates a block-type storage volume, if it is detected that a persistent volume declaration corresponding to the storage volume includes a preset tag, generating a target file including distributed object storage block device information; using the target authentication information to perform a deployment operation of a containerized data import-related component, after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, adding a preset tag to the persistent volume declaration corresponding to the data storage volume; in the process of starting a target container set by a virtual machine instance in a virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes a preset tag, obtaining target encryption information required for mapping the distributed object storage block device from the data storage volume, and mounting the target encryption information to a preset directory; if it is detected that the target authentication information exists in the preset directory, adding the target authentication information to the configuration information of the target virtual machine, and starting the target virtual machine.

[0091] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0092] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0093] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0098] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining target authentication information of the distributed storage system, and storing the target authentication information in the container management system; When the container management system creates a block-type storage volume, if it is detected that the persistent volume declaration corresponding to the storage volume includes a preset label, a target file including distributed object storage block device information is generated; Using the target authentication information to perform a deployment operation of a containerized data import related component, after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, adding the preset tag to the persistent volume declaration corresponding to the data storage volume; In the process of starting the target container set by the virtual machine instance in the virtual machine management platform, if it is detected that the persistent volume declaration corresponding to the data storage volume includes the preset label, the target encryption information required for mapping the distributed object storage block device is obtained from the data storage volume, and the target encryption information is mounted to the preset directory; If it is detected that the target authentication information exists in the preset directory, the target authentication information is added to the configuration information of the target virtual machine, and the target virtual machine is started.

2. The method according to claim 1, characterized in that If it is detected that the target authentication information exists in the preset directory, the method further includes: generating an encrypted file corresponding to the target authentication information, and calling a preset interface to access the storage part in the distributed storage system.

3. The method according to claim 1 or 2, characterized in that: Starting the target virtual machine includes: Traversing the disks to be added to the target virtual machine; When traversing the disk to be added to the target virtual machine, if the block to be mounted is the target file and the distributed object storage block device information in the target file is readable information, add monitor information to the distributed object storage block device information, and add disk information to the target virtual machine in the form of a network disk; After adding the disk information to the target virtual machine, the target virtual machine is started.

4. The method according to claim 1, characterized in that: Storing the target authentication information in a container management system includes: The target authentication information is stored in the form of a first encrypted object in the namespace of the containerized data import-related component of the virtual machine management platform in the container management system, and the target authentication information is added to the configuration information of the containerized data import-related component.

5. The method according to claim 1, characterized in that: After adding the preset tag to the persistent volume declaration corresponding to the data storage volume, the method further includes: Reading the target authentication information and storing the target authentication information in the form of a second encrypted object; The storage part in the distributed storage system is accessed through the target authentication information in the second encrypted object.

6. The method according to claim 1, characterized in that The preset tags include: "cdi.kubevirt.io / storage.directio" = true.

7. The method according to claim 1, characterized in that Mounting the target encrypted information into a preset directory includes: mounting the target encrypted information into the preset directory in a read-only form, wherein the preset directory includes: / etc / ceph.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire target authentication information of a distributed storage system and store the target authentication information in a container management system; A generation module, configured to generate a target file including distributed object storage block device information when the container management system creates a block-type storage volume and if it is detected that a persistent volume declaration corresponding to the storage volume includes a preset label; An adding module is used to use the target authentication information to perform a deployment operation of the containerized data import related components, and after completing the deployment operation, if it is detected that the value type corresponding to the direct input and output in the specification description information corresponding to the data storage volume is true, add the preset label in the persistent volume declaration corresponding to the data storage volume; A processing module, configured to obtain target encryption information required for mapping a distributed object storage block device from the data storage volume if it is detected that the persistent volume declaration corresponding to the data storage volume includes the preset tag during the process of starting the target container set by the virtual machine instance, and mount the target encryption information to a preset directory; The startup module is used to add the target authentication information to the configuration information of the target virtual machine and start the target virtual machine if it is detected that the target authentication information exists in the preset directory.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images