Mirror image data processing method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a mirror image data processing method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that all fragments of a virtual machine mirror image file to be imported with a persistent volume statement in a data source are obtained, and the fragments of the virtual machine mirror image file are uploaded to the data source after the virtual machine mirror image file is sliced and packaged into a container mirror image format; arranging the fragments corresponding to the virtual machine mirror image file to obtain a complete virtual machine mirror image file; and importing the complete virtual machine mirror image file into the persistent volume declaration fragment by fragment. According to the method and the device, the technical problem of pushing failure caused by overtime container mirror image warehouse processing due to overlarge single layer when the virtual machine mirror image packaged in the container mirror image format is pushed in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a mirror data processing method, device, electronic device and non-volatile storage medium. Background Art

[0002] In KubeVirt, the process of using Data Volume (DV) to pull images from the container registry and write them to Persistent Volume Claim (PVC) involves a special Pod, the importer Pod. This Pod is mainly responsible for pulling, converting, and writing images.

[0003] For virtual machine images encapsulated in the OCI (Open Container Initiative) container image format, if a single layer of the pushed virtual machine image is too large, it may cause the container image repository to time out, resulting in the sending client failing to retry. In addition, since Docker does not support single-layer breakpoint resumption downloading in the default mode, it is easy to cause the virtual machine image to retry downloading when the network is poor.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image data processing method, device, electronic device and non-volatile storage medium to at least solve the technical problem of related technologies in which, when pushing a virtual machine image encapsulated in a container image format, a container image repository processing timeout occurs due to a single layer being too large, causing push failure.

[0006] According to one aspect of an embodiment of the present application, a method for processing image data is provided, including: obtaining each slice of a virtual machine image file to be imported into a persistent volume declaration in a data source, wherein the slices of the virtual machine image file are uploaded to the data source after slicing the virtual machine image file and encapsulating it in a container image format; arranging each slice corresponding to the virtual machine image file to obtain a complete virtual machine image file; and importing the complete virtual machine image file into the persistent volume declaration slice by slice.

[0007] Optionally, in the case of planning to import a virtual machine image file into a data source, the method includes: obtaining attribute information of the virtual machine image file, wherein the attribute information includes: file size; in the case that the file size of the virtual machine image file exceeds a preset file size threshold, determining a slice size of the virtual machine image file based on the file size of the virtual machine image file; determining the number of slices that can be obtained after slicing the virtual machine image file according to the slice size, and in the case that the number of slices is not greater than the maximum number of layers supported by the container image, slicing the virtual machine image file according to the slice size; converting the sliced ​​virtual machine image file into a container image format, and importing it into the data source.

[0008] Optionally, converting the sliced ​​virtual machine image file into a container image format includes: determining a slice name corresponding to each slice obtained by slicing the virtual machine image file, wherein the slice name includes: a first character part and a second character part, the first character part is a preset fixed prefix, and the second character part is used to characterize the slicing order between the slices; storing all the slices corresponding to the virtual machine image file in a folder with the same name as the virtual machine image file; adding compilation instructions corresponding to each slice in the folder to the container compilation file, and executing the container compilation file to obtain the container image corresponding to the virtual machine image file, wherein the compilation instructions are used to instruct to add the slices to the container image during the compilation process.

[0009] Optionally, the method includes: in the case where it is planned to use a containerized data importer to import the virtual machine image file in the data source into the persistent volume declaration, determining the import mode of the containerized data importer, wherein the data source includes: an image warehouse; in the case where the import mode is a pod mode, obtaining the image layer information of the image warehouse, and parsing the tar package information corresponding to each image layer to obtain the directory structure corresponding to the image layer; in the case where there is hierarchical directory image metadata information in the directory structure, creating a virtual machine image hole file and verifying the shard file information based on the metadata information, wherein the hierarchical directory image metadata information is used to characterize that the image layer is a shard after the virtual machine image file is sliced, the virtual machine image hole file is a file with pre-allocated space but no content filled, and the verification shard file information is used to characterize the metadata and sequence information of each shard obtained after the virtual machine image file is sliced; based on the virtual machine image hole file and the verification shard file information, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file.

[0010] Optionally, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file, including: determining the slicing order corresponding to each slicing according to the slicing name of the slicing corresponding to the image layer; using multiple processes to concurrently download the image layer corresponding to the slicing in the image repository, and decompressing the corresponding tar package file to obtain each slicing; writing each slicing into the position corresponding to the slicing in the virtual machine image hole file according to the slicing order to obtain a complete virtual machine image file.

[0011] Optionally, the method further includes: determining a process status of each process downloading an image layer; when the process status is a download failure status, determining an image layer corresponding to the download failure status, and re-downloading the image layer.

[0012] Optionally, the method also includes: when the import mode is the node mode, using the service executable file to load the virtual machine image file, and determining whether the virtual machine image file has been sliced; when it is determined that the virtual machine image file has been sliced, determining the corresponding slicing order based on the slicing names of each slicing obtained by slicing; creating a virtual HTTP file, and using the HTTP file to logically connect the slicing in series according to the slicing order to obtain a complete virtual machine image file.

[0013] According to another aspect of an embodiment of the present application, there is also provided an image data processing device, including: a data acquisition module, used to obtain each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slices of the virtual machine image file are uploaded to the data source after slicing the virtual machine image file and encapsulating it into a container image format; a slice reorganization module, used to arrange the slices corresponding to the virtual machine image file to obtain the complete virtual machine image file; and an image writing module, used to import the complete virtual machine image file into the persistent volume declaration slice by slice.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the mirror data processing method is executed when the program is run.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the mirror data processing method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the mirror data processing method when the computer program is executed by a processor.

[0017] In an embodiment of the present application, the method of obtaining each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source is adopted, wherein the slices of the virtual machine image file are uploaded to the data source after the virtual machine image file is sliced ​​and encapsulated into the container image format; the slices corresponding to the virtual machine image file are arranged to obtain a complete virtual machine image file; and the complete virtual machine image file is imported into the persistent volume declaration slice by slice. By slicing the virtual machine image, the original advantages of the OCI container image format are brought into play, and multi-threaded image layers are supported to be pulled from multiple sources, thereby reducing the pressure of a single source and improving transmission efficiency. The technical problem of push failure caused by the container image repository processing timeout due to the large size of a single layer when pushing a virtual machine image encapsulated in the container image format in the related technology is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for image data processing provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a method flow for image data processing provided according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of a virtual machine image sharding process provided according to an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of a process of pulling, converting and writing an image in a pod mode provided according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of a process of pulling, converting and writing an image in a node mode provided according to an embodiment of the present application;

[0024] Figure 6 It is a structural schematic diagram of a mirror data processing device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0028] Kubevirt: is an open source project designed to run and manage virtual machines in a Kubernetes cluster. It encapsulates virtual machines as Kubernetes resources (such as Virtual Machine Instance, VMI), allowing users to leverage the power of Kubernetes to manage virtual machines.

[0029] CDI (Containerized Data Importer): As part of Kubevirt, CDI provides a way to import, manage, and use virtual machine images in a Kubernetes cluster. It simplifies the process of importing virtual machine images and improves the workflow of Kubevirt and its storage.

[0030] OCI Container Image Format: is an open standard that aims to define the layout and format of container images so that they can run on different platforms.

[0031] In the related art, when pushing a virtual machine image encapsulated in the OCI container image format, if a single layer is too large, it may cause the container image repository to time out and the sending client to fail; in addition, since the default Docker mode does not support breakpoint resumption of downloading of a single layer, it is easy to cause the virtual machine image to retry downloading when the network is poor.

[0032] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0033] According to an embodiment of the present application, a method embodiment of mirror data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a mirror data processing method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the mirror data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned mirror data processing method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0039] In the above operating environment, the embodiment of the present application provides a mirror data processing method. Figure 2 is a schematic diagram of a method flow for image data processing provided according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0040] Step S202, obtaining each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slice of the virtual machine image file is uploaded to the data source after slicing the virtual machine image file and encapsulating it into a container image format;

[0041] Step S204, arranging the slices corresponding to the virtual machine image file to obtain a complete virtual machine image file;

[0042] Step S206, import the complete virtual machine image file into the persistent volume declaration slice by slice.

[0043] Through the above steps, by slicing the virtual machine image, policy-driven layered distribution is achieved, that is, the distribution strategy can be formulated according to factors such as node characteristics and network conditions to achieve on-demand distribution of image layers, giving full play to the original advantages of the OCI container image format, supporting multi-threaded pulling of image layers from multiple sources, achieving the purpose of reducing the pressure of a single source and improving transmission efficiency, and thus solving the technical problem of related technologies causing push failure due to container image repository processing timeout caused by a single layer being too large when pushing a virtual machine image encapsulated in the container image format.

[0044] The mirror data processing method in steps S202 to S206 of the embodiment of the present application is further introduced below.

[0045] The embodiment of the present application is based on the original mechanism of Kubevirt CDI to encapsulate virtual machine images in OCI container image format, firstly slices the virtual machine image, and then modifies the code of Kubevirt CDI importer process to realize the writing and reading conversion of the slice file. The following is a detailed introduction.

[0046] First, the slicing process of the virtual machine image file is introduced. The specific steps are as follows.

[0047] In some embodiments of the present application, when it is planned to import a virtual machine image file into a data source, the method includes the following steps: obtaining attribute information of the virtual machine image file, wherein the attribute information includes: file size; when the file size of the virtual machine image file exceeds a preset file size threshold, determining a slice size of the virtual machine image file based on the file size of the virtual machine image file; determining the number of slices that can be obtained after slicing the virtual machine image file according to the slice size, and when the number of slices is not greater than the maximum number of layers supported by the container image, slicing the virtual machine image file according to the slice size; converting the sliced ​​virtual machine image file into a container image format, and importing it into the data source.

[0048] Specifically, Figure 3As shown in the figure, after the virtual machine image file is passed in, the virtual machine image file is first deeply parsed to extract its key attribute information, including file size, type, creation date, etc., to form structured metadata information. Then, according to the preset rules, the virtual machine image file is classified according to its file size, and each category corresponds to an optimal slice size. Among them, the number of slices that can be obtained after slicing must be less than the maximum number of layers (that is, the maximum number of layers supported by the container image). For example, assuming the following rules: for files of 1GB-5GB, the corresponding slice size is 1GB; for files of 5GB-10GB, the corresponding slice size is 2GB; for files larger than 10GB, the corresponding slice size is 5GB; then for a 20GB virtual machine image file, 5GB is selected as its corresponding optimal slice size. This step aims to balance the utilization of storage space and the speed of data retrieval. By dynamically adjusting the slice size, the transmission efficiency can be further optimized under the premise of ensuring file integrity and availability, which is suitable for virtual machine image files of different sizes and enhances the versatility and adaptability of the solution.

[0049] After that, it is determined whether the size of the virtual machine image needs to be sliced. In the scenario where slicing is required, for example, when the file size of the virtual machine image file exceeds the preset file size threshold, the virtual machine image file is sliced ​​according to the slice size. The sliced ​​virtual machine image file is then converted into a container image. The specific steps are as follows.

[0050] In some embodiments of the present application, converting a sliced ​​virtual machine image file into a container image format includes the following steps: determining a slice name corresponding to each slice obtained by slicing the virtual machine image file, wherein the slice name comprises: a first character part and a second character part, the first character part is a preset fixed prefix, and the second character part is used to characterize the slicing order between the slices; storing all slices corresponding to the virtual machine image file in a folder with the same name as the virtual machine image file; adding compilation instructions corresponding to each slice in the folder to the container compilation file, and executing the container compilation file to obtain a container image corresponding to the virtual machine image file, wherein the compilation instructions are used to instruct to add the slices to the container image during the compilation process.

[0051] Specifically, the slices obtained after slicing the virtual machine image file are named, and each slice name is uniformly part.xx, where part. is a fixed prefix (i.e., the first character part mentioned above), and xx is a random name cut in sequence (i.e., the second character part mentioned above), such as part.aa, part.ab, part.ac, etc. Then all the slices corresponding to the virtual machine image file are placed in a folder with the same name as the virtual machine image, and corresponding compilation instructions, such as ADD instructions, are added separately in the container compilation file of the image compilation for each part slice and metadata information, and after the container compilation file is written, the container image is compiled to obtain the virtual machine image file converted into a container image.

[0052] This naming and storage strategy not only makes it easy to identify and manage each shard, but also simplifies the container image building process, facilitating subsequent image import and reorganization. It is suitable for scenarios that require frequent updates and management of image files, such as the rapid construction of development and testing environments and resource management of enterprise-level cloud platforms.

[0053] The following is a further introduction to the process of using the containerized data importer (CDI (Containerized Data Importer)) to pull images from the container image repository and write them into the persistent volume claim PVC.

[0054] In the implementation of this application, CDI pullMethod includes two different import modes, which are introduced below. First, when the import mode is pod (default mode), the specific steps are as follows.

[0055] In some embodiments of the present application, the method includes the following steps: in the case where a containerized data importer is planned to be used to import a virtual machine image file in a data source into a persistent volume declaration, determining an import mode of the containerized data importer, wherein the data source includes: an image warehouse; in the case where the import mode is a pod mode, obtaining the image layer information of the image warehouse, and parsing the tar package information corresponding to each image layer to obtain the directory structure corresponding to the image layer; in the case where there is hierarchical directory image metadata information in the directory structure, creating a virtual machine image hole file and verifying the shard file information based on the metadata information, wherein the hierarchical directory image metadata information is used to characterize that the image layer is a shard after the virtual machine image file is sliced, the virtual machine image hole file is a file with pre-allocated space but no content filled, and the verification shard file information is used to characterize the metadata and sequence information of each shard obtained after the virtual machine image file is sliced; based on the virtual machine image hole file and the verification shard file information, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file.

[0056] In some embodiments of the present application, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file, including the following steps: determining the slicing order corresponding to each slicing according to the slicing name of the slicing corresponding to the image layer; using multiple processes to concurrently download the image layer corresponding to the slicing in the image repository, and decompressing the corresponding tar package file to obtain each slicing; writing each slicing into the virtual machine image hole file in the position corresponding to the slicing according to the slicing order to obtain a complete virtual machine image file.

[0057] Specifically, when pullMethod is in pod mode, the process of importer downloading layer (image layer) is as follows Figure 4 As shown in the figure, in pod mode, the import process actively pulls the layer information (image layer information) in the docker image repository. When the layer layer finds that there are multiple layers, it can first parse the tar package information of each layer to obtain the directory structure of each tar.

[0058] After obtaining all directory structures, if the metadata information of the layered directory image is found, that is, the image layer layer is determined to be a shard after the virtual machine image file is sliced, the shard names of each shard obtained by slicing are used for sorting. And by reading the metadata information, the virtual machine image hole file is generated and the shard file information is checked.

[0059] Then, according to the number of concurrent threads in the defined process file, the image layer is downloaded in a multi-process concurrent manner. The corresponding tar file is decompressed while the layer is downloaded. At the same time, the virtual machine image hole file is sought to the specified position for writing according to the slice order. After all slices are successfully written, the complete virtual machine image file can be obtained. Finally, the entire virtual machine image file that has been downloaded is handed over to the importer for subsequent process conversion.

[0060] The embodiment of the present application can significantly improve the download speed of image files while ensuring the integrity and consistency of image files during the import process through multi-process concurrent downloading and reorganization, and is suitable for scenarios with limited network bandwidth or high pressure on image repository access.

[0061] In addition, the embodiment of the present application will also record the download and write status of each process, and the specific steps are as follows.

[0062] In some embodiments of the present application, the method further includes the following steps: determining the process status of each process downloading the image layer; when the process status is a download failure status, determining the image layer corresponding to the download failure status, and re-downloading the image layer.

[0063] Specifically, if an abnormal situation such as network interruption occurs during the download process, the system can automatically read the progress file, skip the successfully downloaded part, and re-download only the unfinished layer, effectively avoiding duplication of work and improving the robustness and efficiency of the overall process. This error recovery mechanism can effectively cope with network fluctuations or temporary failures of the image repository, ensuring the reliability and continuity of image file import, and is suitable for virtual machine image management scenarios that require high availability and robustness.

[0064] The following is an introduction to the process when the import mode is node mode. The specific steps are as follows.

[0065] In some embodiments of the present application, the method also includes the following steps: when the import mode is the node mode, the virtual machine image file is loaded using the service executable file, and it is determined whether the virtual machine image file has been sliced; when it is determined that the virtual machine image file has been sliced, the corresponding slicing order is determined based on the slicing names of the slicing slices; a virtual HTTP file is created, and the HTTP file is used to logically connect the slicing slices in series according to the slicing order to obtain a complete virtual machine image file.

[0066] Specifically, when the pull method is node mode, the process of the importer server loading the OCI image is as follows: Figure 5 As shown in the figure, in node mode, the corresponding container process (such as dockerd) actively pulls the image file. In this mode, the impoer pod will load the server executable file (service executable file). During the server operation, first check whether the virtual machine image file has been sliced. If it is determined that the virtual machine image file has been sliced, the system will sort all the slice files, that is, arrange them in ascending order according to the serial number in the slice name of each slice to obtain the slice order, thereby ensuring the continuity and integrity of the data.

[0067] Then, using the size information of the slice file, a virtual HTTP file is simulated to create a virtual file. Through this file, the slices are logically connected in series according to the slice order to form a complete image file view. After that, through the HTTP service, the system can provide the slice order of each slice to the importer pod, so that it can read these slice files like reading a single file, and the importer pod can request specific slices on demand without downloading the entire image file.

[0068] With this method, the efficiency of the HTTP protocol can be used to achieve rapid reorganization of image files. It is suitable for scenarios that require the use of existing network protocols and infrastructure, such as optimization of enterprise internal networks and integration of public cloud services.

[0069] The present application scheme can reduce the impact of network delay. After the container virtual machine image is transformed into fragmented download, data can be downloaded from multiple sources at the same time. Even if a source is delayed or fails, other sources can still continue to transmit, thereby reducing the overall download time. When it comes to uploading virtual machine images, if a single layer is too large, it will also cause the container repository to take too long to verify the integrity. The present application scheme can improve the success rate of uploading large images; it fully utilizes the layering capability of the original OCI container image and integrates it with the original container image mode; it enhances security. Fragmented download can protect the security of each fragment through encryption and authentication mechanisms, even if some fragments are intercepted during network transmission; fragmented download makes the management and update of images more flexible, and it is possible to update only part of the image layer; it enhances fault tolerance. If a fragment fails during the download process, only this specific fragment needs to be re-downloaded, without re-downloading the entire image, which greatly improves the fault tolerance and efficiency of the download process; it improves download speed and efficiency. By dividing the container virtual machine image into multiple small pieces for download, the advantages of multi-threaded or multi-connection download can be fully utilized, thereby significantly improving the download speed. This is especially important for environments with poor network conditions or limited bandwidth; Support large-scale deployment: For large-scale multiple Kubernetes kubevirt clusters, shard download can ensure that image downloads can be completed quickly and stably even in high-concurrency scenarios, thereby accelerating the startup and deployment of virtual machine instances.

[0070] According to an embodiment of the present application, an embodiment of a mirror data processing device is also provided. Figure 6 Schematic diagram of a mirror data processing device provided according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0071] A data acquisition module 60 is used for the data acquisition module to obtain each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slice of the virtual machine image file is uploaded to the data source after slicing the virtual machine image file and encapsulating it into a container image format;

[0072] A fragment reorganization module 62, used for arranging the fragments corresponding to the virtual machine image file to obtain a complete virtual machine image file;

[0073] The image writing module 64 is used to import the complete virtual machine image file into the persistent volume declaration slice by slice.

[0074] Optionally, in the case of planning to import a virtual machine image file into a data source, the image data processing device is used to: obtain attribute information of the virtual machine image file, wherein the attribute information includes: file size; when the file size of the virtual machine image file exceeds a preset file size threshold, determine the slice size of the virtual machine image file based on the file size of the virtual machine image file; determine the number of slices that can be obtained after slicing the virtual machine image file according to the slice size, and when the number of slices is not greater than the maximum number of layers supported by the container image, slice the virtual machine image file according to the slice size; convert the sliced ​​virtual machine image file into a container image format, and import it into the data source.

[0075] Optionally, converting the sliced ​​virtual machine image file into a container image format includes: determining a slice name corresponding to each slice obtained by slicing the virtual machine image file, wherein the slice name includes: a first character part and a second character part, the first character part is a preset fixed prefix, and the second character part is used to characterize the slicing order between the slices; storing all the slices corresponding to the virtual machine image file in a folder with the same name as the virtual machine image file; adding compilation instructions corresponding to each slice in the folder to the container compilation file, and executing the container compilation file to obtain the container image corresponding to the virtual machine image file, wherein the compilation instructions are used to instruct to add the slices to the container image during the compilation process.

[0076] Optionally, the image data processing device is used to: determine the import mode of the containerized data importer when it is planned to use a containerized data importer to import the virtual machine image file in the data source into the persistent volume declaration, wherein the data source includes: an image warehouse; when the import mode is a pod mode, obtain the image layer information of the image warehouse, and parse the tar package information corresponding to each image layer to obtain the directory structure corresponding to the image layer; when there is hierarchical directory image metadata information in the directory structure, create a virtual machine image hole file and verify the shard file information based on the metadata information, wherein the hierarchical directory image metadata information is used to characterize the image layer as a shard after the virtual machine image file is sliced, the virtual machine image hole file is a file with pre-allocated space but no content filled, and the verification shard file information is used to characterize the metadata and sequence information of each shard obtained after the virtual machine image file is sliced; based on the virtual machine image hole file and the verification shard file information, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file.

[0077] Optionally, the sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file, including: determining the slicing order corresponding to each slicing according to the slicing name of the slicing corresponding to the image layer; using multiple processes to concurrently download the image layer corresponding to the slicing in the image repository, and decompressing the corresponding tar package file to obtain each slicing; writing each slicing into the position corresponding to the slicing in the virtual machine image hole file according to the slicing order to obtain a complete virtual machine image file.

[0078] Optionally, the image data processing device is further used to: determine the process status of each process downloading the image layer; when the process status is a download failure status, determine the image layer corresponding to the download failure status, and re-download the image layer.

[0079] Optionally, the image data processing device is also used to: when the import mode is the node mode, use the service executable file to load the virtual machine image file, and determine whether the virtual machine image file has been sliced; when it is determined that the virtual machine image file has been sliced, determine the corresponding slicing order based on the slicing names of each slicing obtained by slicing; create a virtual HTTP file, and use the HTTP file to logically connect the slicing in series according to the slicing order to obtain a complete virtual machine image file.

[0080] It should be noted that the various modules in the above-mentioned image data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0081] It should be noted that the mirror data processing device provided in this embodiment can be used to execute Figure 2 The mirror data processing method shown, therefore, the relevant explanations and descriptions of the above-mentioned mirror data processing method are also applicable to the embodiments of the present application and will not be repeated here.

[0082] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following image data processing method by running the computer program: obtaining each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slices of the virtual machine image file are uploaded to the data source after the virtual machine image file is sliced ​​and encapsulated into a container image format; arranging each slice corresponding to the virtual machine image file to obtain a complete virtual machine image file; and importing the complete virtual machine image file into the persistent volume declaration slice by slice.

[0083] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image data processing method described in each embodiment of the present application: obtaining each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slices of the virtual machine image file are uploaded to the data source after the virtual machine image file is sliced ​​and encapsulated into a container image format; arranging each slice corresponding to the virtual machine image file to obtain a complete virtual machine image file; and importing the complete virtual machine image file into the persistent volume declaration slice by slice.

[0084] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0085] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0090] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A mirror data processing method, characterized in that: include: Obtaining each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slice of the virtual machine image file is uploaded to the data source after slicing the virtual machine image file and encapsulating it into a container image format; Arranging the slices corresponding to the virtual machine image file to obtain a complete virtual machine image file; Import the complete virtual machine image file into the persistent volume declaration slice by slice.

2. The mirror data processing method according to claim 1, characterized in that: In the case where it is planned to import the virtual machine image file into the data source, the method further includes: Obtaining attribute information of the virtual machine image file, wherein the attribute information includes: file size; When the file size of the virtual machine image file exceeds a preset file size threshold, determining a slice size of the virtual machine image file according to the file size of the virtual machine image file; Determine the number of slices that can be obtained after slicing the virtual machine image file according to the slice size, and slice the virtual machine image file according to the slice size when the number of slices is not greater than the maximum number of layers supported by the container image; The sliced ​​virtual machine image file is converted into a container image format and imported into the data source.

3. The mirror data processing method according to claim 2, characterized in that: Converting the sliced ​​virtual machine image file into a container image format includes: Determine a slice name corresponding to each slice obtained by slicing the virtual machine image file, wherein the slice name includes: a first character part and a second character part, the first character part is a preset fixed prefix, and the second character part is used to represent the slicing order between each slice; Storing all the slices corresponding to the virtual machine image file in a folder with the same name as the virtual machine image file; Add compilation instructions corresponding to each of the shards in the folder to the container compilation file, and execute the container compilation file to obtain the container image corresponding to the virtual machine image file, wherein the compilation instructions are used to instruct the shards to be added to the container image during the compilation process.

4. The mirror data processing method according to claim 1, characterized in that: The method comprises: In the case where it is planned to use a containerized data importer to import the virtual machine image file in the data source into the persistent volume declaration, determining an import mode of the containerized data importer, wherein the data source includes: an image repository; When the import mode is pod mode, the image layer information of the image warehouse is obtained, and the tar package information corresponding to each image layer is parsed to obtain the directory structure corresponding to the image layer; In the case where there is hierarchical directory image metadata information in the directory structure, a virtual machine image hole file and verification slice file information are created according to the metadata information, wherein the hierarchical directory image metadata information is used to characterize that the image layer is a slice after the virtual machine image file is sliced, the virtual machine image hole file is a file with pre-allocated space but no content filled, and the verification slice file information is used to characterize the metadata and sequence information of each slice obtained after the virtual machine image file is sliced; According to the virtual machine image hole file and the verification slice file information, the sliced ​​virtual machine image file is reorganized to obtain the complete virtual machine image file.

5. The mirror data processing method according to claim 4, characterized in that: The sliced ​​virtual machine image file is reorganized to obtain a complete virtual machine image file including: Determine the slicing order corresponding to each shard according to the shard name of the shard corresponding to the image layer; Using multiple processes to concurrently download the image layer corresponding to the shard in the image repository, and decompressing the corresponding tar package file to obtain each of the shards; According to the slicing order, each of the slices is written into the position corresponding to the slice in the virtual machine image hole file to obtain the complete virtual machine image file.

6. The mirror data processing method according to claim 5, characterized in that: The method further comprises: Determine the process status of each process downloading the image layer; When the process status is a download failure status, the image layer corresponding to the download failure status is determined, and the image layer is downloaded again.

7. The mirror data processing method according to claim 4, characterized in that: The method further comprises: When the import mode is the node mode, the virtual machine image file is loaded by using the service executable file, and it is determined whether the virtual machine image file has been sliced; In the case where it is determined that the virtual machine image file has been sliced, determining the corresponding slicing order according to the slice names of the slices obtained by slicing; A virtual HTTP file is created, and the HTTP file is used to logically connect the slices in series according to the slice sequence to obtain a complete virtual machine image file.

8. A mirror data processing device, characterized in that: include: A data acquisition module, used to acquire each slice of the virtual machine image file to be imported into the persistent volume declaration in the data source, wherein the slice of the virtual machine image file is uploaded to the data source after slicing the virtual machine image file and encapsulating it into a container image format; A fragment reorganization module, used for arranging the fragments corresponding to the virtual machine image file to obtain a complete virtual machine image file; The image writing module is used to import the complete virtual machine image file into the persistent volume declaration slice by slice.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the mirror data processing method according to any one of claims 1 to 7 when running.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the mirror data processing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the mirror data processing method according to any one of claims 1 to 7 are implemented.

Images