A method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform

Abstract

The present invention relates to the technical field of computer cloud computing, and more specifically to a method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform. By introducing a layer-based mirror image management technology, the mirror image management granularity is adjusted from the mirror image unit to the mirror image layer unit. The mirror image layer stores specific content and uses a unique fingerprint identifier. The mirror image can be separated and spliced on a per-layer basis as needed, providing technical support for mirror image cropping, rapid generation of heterogeneous mirror images, and mirror image preheating. The layer-based mirror image management technology stores the mirror image layer information and the relationships between mirror image layers in a mirror image metadata warehouse, making zero intrusion into the existing mirror image specifications and being compatible with existing mirror images, solving problems such as large mirror image storage space occupation, low mirror image distribution efficiency, slow mirror image startup speed, the need to remanufacture mirror images when adapting to new architectures, and high maintenance costs for multi-architecture mirror images.

Description

Technical Field

[0001] The present invention relates to the technical field of computer cloud computing, and more specifically, to a method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform, which is mainly used for lightweight cloud platforms and cloud-native applications. Background Art

[0002] As a new type of technology system, cloud-native technology has become the future development direction of cloud computing after rapid development since it was first proposed in 2013. The establishment of the CNCF Foundation marks the official entry of cloud-native technology into the public eye. With the continuous growth of the cloud-native ecosystem in recent years, all major cloud providers have joined the CNCF Foundation, and the cloud-native ecosystem is flourishing.

[0003] Cloud-native technology is a set of cloud technology product systems built on the basis of containers, combined with technologies such as microservices and DevOps. Container technology, as the foundation of cloud-native technology, plays a crucial role in the cloud-native system.

[0004] As the carrier of containers, images contain the container running environment. The Docker image specification is the de facto standard for container images. An image consists of image layers. A complete image usually contains multiple image layers. Docker provides the docker build command for image production. Users edit the Dockerfile according to the Docker image syntax specification, write the instructions for image production into the Dockerfile, and use the Dockerfile as the input to run the docker build command to complete image production. Specific instructions in the Dockerfile will generate an image layer, such as ADD and RUN instructions. These instructions are used to complete operations such as copying application installation packages and dependencies into the image, running scripts to complete application deployment, and modifying application configurations. Users usually do not need to build images from scratch. More often, new images are made based on existing images. Image derivation makes the images have a parent-child relationship. The child image will inherit all the image layers of the parent image and add new image layers on the basis of the parent image's image layers to implement its own functions.

[0005] An image is the basic unit managed by Docker. After a well-made image is exported, it needs to be distributed to the production environment for deployment and startup. The exported image contains all image layers. To improve the efficiency of image distribution, Docker provides Docker Registry to support the image distribution service. Docker Registry is the image repository. After an image is imported into Docker Registry, the Docker client can access Docker Registry through the network to retrieve and download the image. Before an image runs, it must first be downloaded to the local image repository. If the local image repository does not have the image to be run, the Docker service will access Docker Registry to retrieve and download the target image.

[0006] The self - contained feature of images (an image contains the application and the environment required for the application to run. In addition to the application itself, all the required dependency libraries, configurations, tools, etc. for the application to run are included in the image) and the derivative relationship between images significantly increase the image size. For example, AI model images based on CUDA are usually over 5G, and the size of some images can reach over 10G. Big data - related images such as HDFS and YARN are usually around 2G, and the basic image containing the JAVA runtime environment is around 700M.

[0007] Container technology belongs to lightweight virtualization technology. Before starting an instance on a container cloud platform, the image needs to be uploaded from the local machine to the image repository and then distributed to the running nodes. Once the image is ready, the container can be started in milliseconds. After testing, the time for uploading and downloading images on average accounts for more than 90% of the container deployment time. Oversized images occupy a large amount of network bandwidth. Especially in restricted internal networks and secure networks, usually 10 or more containers run on a single physical machine. As the number of running containers increases and the size of a single image increases, the space required to store images surges.

[0008] The root cause of the above problems is due to the existing image technology focusing on the management scheme at the image granularity. The main problems are as follows:

[0009] 1. Large occupation of image storage space and low image distribution efficiency:

[0010] The large size of the image is, on the one hand, determined by the self - contained feature of the image. The image needs to package the application and all the environments required for the application to run, including the OS basic libraries, various business libraries, and dependent components, and these environments themselves occupy a considerable amount of storage.

[0011] Another reason for the large mirror image is caused by the mirror image layer. The mirror image layers are in a stacked relationship. The upper layer expands on the basis of the lower layer. Deleting or modifying files is achieved by appending a new layer and writing a deletion mark or adding a new file in the new layer. The deletion mark and the new file block all versions of the underlying file. The original file and all file copies still remain in the lower layer. Even within the same mirror image layer, the existence of garbage files will also cause an increase in the size of the mirror image layer. For example, the application installation package copied into the mirror image layer and the temporary files generated during application deployment become garbage data after the application deployment is completed. These data significantly increase the mirror image size.

[0012] The large mirror image seriously reduces the efficiency of mirror image import, export, and distribution, occupies valuable storage and network resources, and prolongs the application deployment startup time.

[0013] 2. Unfriendly support for heterogeneous platforms and domestic hardware:

[0014] Due to the different CPU architectures of heterogeneous platforms, their instruction sets and binary files are also incompatible. When supporting each new architecture, it is necessary to re-create the mirror image corresponding to that architecture. Even if the application itself has not changed, the production, storage, testing, and maintenance of the new mirror image require additional costs. From the perspective of the application, these consumptions should be avoided and are meaningless. The multi-architecture mirror image further exacerbates the expansion of the mirror image size. When distributing the mirror image, it is necessary to support all possible architectures. Therefore, the mirror images of relevant architectures have to be distributed to the site, stored in the mirror image repository, and kept updated and maintained. The popularization and application of domestic hardware have made this situation even more severe.

[0015] 3. Slow mirror image pulling and startup speed, and low system response efficiency:

[0016] Large-scale distributed systems, especially online systems, have very high real-time requirements. Some applications carried by the platform have very strong dynamics and real-time characteristics, and it is necessary to dynamically start and stop containers according to the application load in order to quickly respond to user requests. For example, for some dynamic analysis tasks, after the user issues a request, it must be quickly started, and after completing the calculation and statistics of the results, it is immediately destroyed to release resources. Such high-performance platforms have very high requirements for the container startup efficiency. In the existing technical system, the mirror image pulling starts only after the node receives the container startup request, which greatly reduces the system response speed and cannot support the operation of high-performance systems.

[0017] The above problems can be solved from the following aspects:

[0018] 1. Optimize the mirror image production process. By a reasonable mirror image production process, reduce the mirror image layer. Fewer layers help reduce the mirror image size;

[0019] 2. Adopt a layer-based management strategy for hierarchical optimization. Divide the image layers into common image layers, file copy layers, and business image layers. Each layer stores specific content. The common image layer saves the basic running environment, and the business image layer stores the application. Identify the capabilities of the image layers through function tags and record the image layer information in the image metadata repository, such as the JAVA capability layer, PYTHON capability layer, etc. Each capability can be further subdivided according to characteristics and versions, such as JAVA1.8, JAVA15, etc. When uploading an image to the image repository, analyze the image metadata, strip the image layer information, and synchronously update the image metadata repository; develop a layer-based splicing technology that can splice several image layers into a finished image that meets the requirements according to needs. The layer-based management strategy is the basis for image reduction and preheating;

[0020] 3. Image layer trimming. Delete the intermediate files generated during image production, such as the installed packages and temporary files after deployment, to reduce the size of the image layer;

[0021] 4. Common image layer trimming. Trim the common image layer inside the image to avoid repeated distribution of the same basic environment;

[0022] 5. Business image layer aggregation. After image layer trimming and common image layer trimming, merge the remaining image layers to form a business image layer (these image layers save the application and its direct dependencies, so they can be merged). This solution is of great significance for the rapid generation of heterogeneous platform images. For applications developed based on interpreted languages such as JAVA, PYTHON, and SHELL, after adopting this solution, images for the target architecture can be quickly constructed based on the business image layer and the common image layer of the target architecture.

[0023] Currently, the industry's solutions to the above problems are limited. Existing solutions mainly focus on accelerating image pulling, such as accelerating image distribution through P2P technology. Since there are multiple reasons affecting image distribution efficiency and startup speed, and image pulling is just one of them, simply accelerating image pulling through P2P technology has limited impact on container startup speed and cannot solve the first-time distribution of images and applications in intranets and secure network environments (distributing images from the development environment to the intranet, and overly large images affect distribution efficiency).

[0024] P2P technology divides files into blocks, and all hosts pulling images can act as download servers. When a certain node pulls a block, it can provide block download services externally to make the most of network bandwidth and avoid network congestion. From the perspective of overall resource utilization, P2P technology can make full use of network resources, enabling more nodes to participate in data distribution to improve distribution efficiency. However, P2P technology still has the following limitations:

[0025] Although P2P can effectively utilize network bandwidth resources and make the overall bandwidth utilization relatively balanced, it is still limited by the total bandwidth. For example, when a seed node downloads data blocks from a repository, if the network bandwidth is insufficient, the data distribution efficiency will be reduced;

[0026] The amount of data downloaded by each node remains unchanged. The amount of data downloaded by a node will not decrease due to the adoption of P2P technology. For example, if node NodeA downloads a certain image for the first time and the size of the image is 2G, then the network traffic generated by NodeA's final download of the image is at least 2G. It's just that after adopting P2P, some data comes from other nodes, optimizing the central bottleneck problem;

[0027] The improvement of the mirror startup speed is limited. As mentioned before, mirror pulling is only one of the many links that affect the mirror startup speed, and the optimization of the mirror pulling speed has limited improvement on the system response speed.

[0028] Therefore, the present invention proposes a method for layer-based mirror trimming, rapid production of multi-architecture mirrors, and mirror preheating to solve the problems existing in the prior art solutions and accelerate the popularization and application of container technology. Summary of the Invention

[0029] To solve the above technical problems, the present invention provides a method for mirror trimming and multi-architecture mirror construction based on a cloud platform. By introducing layer-based mirror management technology, the mirror management granularity is adjusted from the mirror unit to the mirror layer unit. The mirror layer stores specific content and uses a unique fingerprint identifier to solve problems such as large mirror storage space occupation, low mirror distribution efficiency, slow mirror startup speed, the need to remake the mirror when adapting to a new architecture, and high maintenance costs of multi-architecture mirrors in the cloud native environment.

[0030] The specific technical solution of the present invention is as follows:

[0031] A method for mirror trimming and multi-architecture mirror construction based on a cloud platform, the mirror trimming and multi-architecture mirror construction method includes the following steps:

[0032] S1: Introduce layer-based management technology, and divide the image layers into a common image layer for storing the basic running environment, a file copy layer for storing temporary files, and a business image layer for storing applications. Among them, the common image layer can be further subdivided, such as the JDK layer for storing the JAVA running environment, the LIB layer for storing dynamic libraries, etc. Each image layer has a unique identifier and a function label describing the layer's capabilities. These information are recorded in the image metadata repository to facilitate the retrieval and management of image layers. At the same time, the means of image production and upgrade are more flexible and diverse. Adding a new image layer on the basis of the parent image is no longer the only means of image production and upgrade. Users can quickly generate the target image based on the existing image layers. They only need to list the functions required by the target image through a configuration file. The cloud platform retrieves the image metadata repository according to the function label and stitches the matching image layers into the target image. The upgrade of the image is also more efficient. When an image layer is updated, the changed image layer is deleted from the original image and replaced with a new image layer, and the metadata of the image is modified at the same time to achieve in-place upgrade of the image.

[0033] S2: Image layer pruning. When generating an image layer, identify and recognize the temporary data in the image layer and delete it when generating the image, thereby reducing the size of a single-layer image.

[0034] S3: Common image layer pruning. Identify and recognize the common image layer in the image, and delete the common image layer to reduce the image size.

[0035] S4: Fast generation of heterogeneous images. Scan and identify the business image layer. When adapting to a new architecture, quickly generate an image of another architecture based on one architecture according to the attributes of the business image layer. For multi-architecture images, only one business image layer needs to be maintained, reducing the maintenance cost.

[0036] S5: Image preheating. Through preheating mechanisms such as image cycle preheating, trigger preheating, and prediction preheating, move the image pulling time forward, reduce the image startup time, obtain a good response speed, thereby improving the response speed of the distributed system and achieving fast startup of containers.

[0037] As a further improvement of this solution, in S1, the layer-based image management technology divides the image layers into a basic environment layer, a dependency layer, a file copy layer, and a business layer according to functions:

[0038] Basic image layer: Store the basic running environment related to the OS and architecture.

[0039] Dependency layer: Store the direct dependencies during application runtime, including the runtime environment, dynamic libraries, etc., such as the JAVA environment, PYTHON environment, and encryption and decryption libraries relied on by security applications.

[0040] Common image layer: The base image layer and the dependency layer are collectively referred to as the common image layer. The common image layer belongs to common resources and can be shared by multiple images;

[0041] File copy layer: The content saved in this layer is temporary and plays an auxiliary role during image creation. After the image creation is completed, the data in this layer is no longer needed; such as application installation packages. This solution requires that the one-time data and temporary data used during image creation be saved in the / tmp directory. Any non-empty image layer in the / tmp directory belongs to the file copy layer. Ideally, if all temporary files are copied to the / tmp directory in one command during image creation, the image will have only one file copy layer;

[0042] Business layer: Saves the data and configurations of the application itself.

[0043] It should be noted that the base environment layer, the dependency layer, the common image layer, the file copy layer, and the business layer classify the image layers from a functional perspective. It is a classification method and is a general term for a type of image layer with the same attributes. Each classification can contain multiple image layers. For example, the dependency layer can consist of a JAVA layer that saves the JAVA running environment, a PYTHON layer that saves the PYTHON running environment, and an encryption layer that saves encryption and decryption algorithms and libraries. Specific images do not necessarily contain all functional layers. Whether an image contains a specified functional layer and how many image layers each type of functional layer specifically contains vary depending on the image. For example, the base image does not contain the business layer.

[0044] As a further improvement of this solution, in S1, an image metadata repository is introduced to support layer-based image management technology. The image metadata repository saves image metadata information, including image information and image layer information, which describe the composition and attributes of the image;

[0045] The information saved in the image metadata repository is as follows:

[0046] Image information: Image name, version, architecture, generation time, included image layers, and the relationship between image layers;

[0047] Basic information of the image layer: Image layer name, version, architecture, generation time, and whether the image layer is trimmed;

[0048] Image layer fingerprint: Image layer fingerprint information, which can uniquely identify the image layer;

[0049] Function label of the image layer: A label that describes the function of the image layer, such as JAVA1.8 running environment, PYTHON3.6 running environment;

[0050] Image layer type: The type of the image layer (base environment layer, dependency layer, file copy layer, and business layer).

[0051] In addition, the mirror metadata repository supports indexing mirror layers by fingerprint, architecture, and functional tags.

[0052] To be compatible with existing specifications, the above metadata information (referred to as extended information) is written into the Labels field in the json of the mirror layer metadata file in the form of KEY-VALUE pairs. The KEY identifies the attribute name (such as the mirror layer fingerprint, mirror layer functional tag, mirror layer type, mirror layer architecture), and the VALUE identifies the attribute value. As mentioned before, a mirror is composed of mirror layers, and each mirror layer contains three files: layer.tar, json, and VERSION. layer.tar is the package of the current mirror layer file, VERSION records the specification version information, and json contains the mirror layer attributes and configurations. Recording the extended information in the Labels field of the json file can ensure the compatibility of the specifications. Mirror runtime components such as docker will treat the Labels field as ordinary tags, and these tags only have special meanings in this solution.

[0053] The metadata information is written through a mirror scanner. For a mirror made according to the above mirror layer classification specification, the mirror scanner can add metadata such as the mirror layer fingerprint, functional tag, and type to the mirror layer. For a third-party mirror, the metadata can be manually specified and written through the mirror scanner.

[0054] When a mirror is uploaded, the mirror layer json file is parsed to extract the metadata information and add it to the mirror metadata repository; when a mirror is exported, the information stored in the mirror metadata repository is written back into the mirror again.

[0055] The mirror metadata repository supports the following retrieval methods:

[0056] Fingerprint: Uniquely match the mirror layer by fingerprint;

[0057] Architecture: Match the mirror layer by architecture. There are usually multiple mirror layers with the same architecture, so architecture is often used for advanced filtering;

[0058] Functional tag: Match the mirror layer by functional tag. For example, the JAVA1.8 functional tag can match all mirror layers that contain the JAVA1.8 runtime environment under all architectures;

[0059] Type: Match the mirror layer by type, such as listing all business layer mirrors;

[0060] Mirror: Match the mirror by mirror name and tag, which is usually used to further retrieve mirror information according to the mirror name, such as the mirror layers included in the mirror, the mirror layer type, etc.;

[0061] Three-party images usually do not have extended information. Therefore, specific image layers can be retrieved from three-party images through fingerprints, all image layers included in the image can be retrieved through the image name and tag, or images containing the image layer can be retrieved based on the image layer information. For images that follow this specification, in addition to the above methods, they can also be retrieved through fingerprints, function tags, and types. For example, all image layers containing the JAVA1.8 runtime environment under all architectures can be retrieved through the JAVA1.8 tag. This retrieval method is usually used for the rapid generation of heterogeneous architecture images. For example, the corresponding image layer under the ARM architecture can be matched through the JAVA1.8 tag of the X86 version.

[0062] As a further improvement of this solution, in S2, as Figure 1 shown, the in-image layer trimming is used to trim off the file copy layer, save the disposable data and temporary data used when making the image in the / tmp directory. After the image is made, run the image layer trimmer to delete the data in the / tmp directory and update the image metadata information. The image may contain multiple file copy layers, and the image layer trimmer processes all image layers in sequence to empty the / tmp directory of the image layer.

[0063] As a further improvement of this solution, in S3, the common image layer includes the base image layer and the dependency layer. These image layers can be referenced by multiple images. To avoid duplicate data saving and reduce data redundancy, they need to be trimmed. The trimming of the common image layer is completed by the image layer trimmer. The trimming process of the common image layer is as Figure 2 shown. The image layer trimmer extracts the metadata information of the image to be trimmed, compares the fingerprint of each image layer that makes up the image with the fingerprint saved in the image metadata repository. If the image layer exists in the image metadata repository and the image layer type is the base image layer or the dependency layer, then it is trimmed; otherwise, the current image layer is skipped.

[0064] As a further improvement of this solution, as Figure 3 shown, the trimming of the common image layer depends on the image metadata repository. When the image layer type saved in the image metadata repository is the common image layer, the corresponding image layer can be trimmed. There are the following two ways to update the image metadata repository:

[0065] Automatic update: Update the image metadata repository when uploading the image. If the uploaded image is the base image, then all image layers are marked as the base image layer; otherwise, run the image layer scanner to dynamically identify the image layer type according to the built-in rules. The working principle of the image scanner will be described in the next subsection.

[0066] Manual update: The administrator manually writes the image layer information into the image metadata repository. This method can correct the situation where the image layer scanner misidentifies during automatic update, and can also be used to batch add three-party images.

[0067] The timing of cropping the common image layer is relatively flexible, and there are the following methods:

[0068] Cropping at image generation: After the image is made, run the image layer cropper to crop the common image layer;

[0069] Cropping at image upload: When the image is uploaded to the image repository, run the image layer cropper to crop the common image layer;

[0070] Manual trigger: Manually run the image layer cropper to crop the common image layer.

[0071] When the image cropper runs, it needs to access the image metadata repository. If the image metadata repository is inaccessible, the image cropper will report an error and exit;

[0072] After testing, under the superposition effect of cropping within the image layer and cropping the common image layer, the image import speed in typical scenarios is increased by 40%, and the image size is reduced by 65%.

[0073] As a further improvement of this solution, in S4, the heterogeneous image is quickly generated based on the image layer dynamic stitching technology. The image layer dynamic stitching technology is applicable to architecture-unrelated and semi-architecture-related applications. For architecture-unrelated applications, their architecture relevance is shielded by the base image layer and the dependency layer. By merging the architecture-unrelated business layer and the corresponding base image layer and dependency layer under the target architecture in the order of the image layers saved in the image metadata, the image of the target architecture can be generated;

[0074] Semi-architecture-related applications contain some strongly architecture-related files. When generating the image of the target architecture, the corresponding files under the target architecture need to be provided to reduce the system complexity.

[0075] Images have architecture relevance. The libraries and binaries contained inside the image are compiled into the instruction set of the target architecture and can only be started and run on the CPUs of the corresponding architecture. The clusters in the production environment usually include multiple CPU architectures. Common architectures include X86, ARM, Haiguang, Kunpeng, Feiteng, Loongson, and Shenwei. When deploying services, it is necessary to build images of multiple architectures to adapt to servers of different architectures.

[0076] Multi-architecture images pose challenges to image production, maintenance, update, and upgrade. Even if the application itself has not made any changes, in order to support multi-architecture deployment, it is necessary to produce and maintain images of multiple architectures.

[0077] According to the relationship between the application and the CPU architecture, applications can be divided into three categories:

[0078] Architecture-independent: The application does not need to be aware of the CPU architecture. For interpreted languages such as JAVA, PYTHON, and SHELL, taking JAVA as an example, after the source code is compiled by the javac command, the resulting JAVA bytecode file is independent of the CPU architecture and can run on any device with a JAVA runtime environment. The architecture dependence is shielded by the JAVA runtime environment. JAVA provides different JVMs for various architectures (such as X86 and ARM). At runtime, the JVM converts the bytecode into machine code for execution. Therefore, for JAVA applications, they themselves are not aware of the CPU architecture and belong to architecture-independent applications, and JAVA belongs to architecture-independent languages;

[0079] Highly architecture-dependent: The application is bound to the CPU architecture and can only run on the target architecture CPU. For example, compiled languages such as C and C++ are compiled, assembled, and linked to form binaries for a specific architecture and can only run on the target CPU architecture. C and C++ are also called architecture-dependent languages;

[0080] Semi-architecture-dependent: The application main body is developed in architecture-independent languages such as JAVA and PYTHON and carries a small amount of binary libraries developed in architecture-dependent languages such as C and C++. For example, the HDFS application is mainly developed in JAVA, and the encryption and decryption algorithms it depends on are developed in C language and encapsulated into dynamic link libraries. Such applications must have libraries for the corresponding architecture when running.

[0081] There are a large number of architecture-independent and semi-architecture-dependent applications in the production environment, and optimizing such applications has practical significance.

[0082] We call the layer remaining after in-mirror-layer trimming and common mirror-layer trimming the business layer. In an ideal state, there is only one layer in the business layer. If there are multiple business layers, they can be merged into one layer. The merging of the business layer is completed by the mirror-layer merger, and the merging logic is as follows:

[0083] Starting from the topmost mirror layer, process each mirror layer in the top-down order. For each mirror layer, unzip the mirror-layer files to the working directory. When unzipping, the files in the lower layer cannot overwrite the files with the same name in the upper layer, that is, the files in the upper layer have a higher priority than the files with the same name in the lower layer; at the same time, process the deletion marks. When there is a deletion mark for a certain file in the upper layer, all files with the same name in the lower layer are blocked, and these files will be skipped during unzipping. Finally, compress the files in the working directory into one mirror layer and update the metadata.

[0084] The merging of the business layer will further reduce the number of mirror layers and shrink the mirror size. After the merging of the business layer is completed, the common mirror layer of the mirror has been trimmed, and the mirror size has been significantly reduced. The storage occupied by the mirror is usually only about 20% - 30% of the original mirror, with very high distribution efficiency.

[0085] After the business layer is merged, run the image layer scanner. Identify the relevance of the business layer architecture based on the file types in the business layer. The scanning logic of the image scanner is as follows:

[0086] If the image layer has already been labeled, skip this scan;

[0087] Scan all binary files in the image layer and determine the file types. If all binary file types in the image layer are related to the architecture, label the current image layer as strongly architecture-related, and write the image layer type and architecture information to the Labels field in the image layer metadata file json, and end the scan of the current image layer;

[0088] If all binary file types in the image layer are not related to the architecture, i.e., applications developed in languages such as JAVA, PYTHON, SHELL, etc., label the current image layer as not architecture-related, and write the image layer type and architecture information to the Labels field in the image layer metadata file json, and end the scan of the current image layer;

[0089] If the image layer contains both architecture-related files and architecture-unrelated files, and the total number of architecture-related files does not exceed the configured upper limit (to simplify the system, when the number of strongly architecture-related files in a semi-architecture-related application exceeds the threshold, it is converted into a strongly architecture-related application), label the current image layer as semi-architecture-related, and write the image layer type and architecture information to the Labels field in the image layer metadata file json, and end the scan of the current image layer; if the total number of architecture-related files exceeds the configured upper limit, label the current image layer as strongly architecture-related;

[0090] If the image layer only contains text files such as configuration files, label the current image layer as not architecture-related, and write the image layer type and architecture information to the Labels field in the image layer metadata file json, and end the scan of the current image layer.

[0091] After the scan is completed, synchronously update the information saved in the image metadata repository.

[0092] The mirror layer dynamic splicing technology is applicable to architecture-unrelated and semi-architecture-related applications. For architecture-unrelated applications, their architecture relevance is shielded by the base mirror layer and the dependency layer. By merging the architecture-unrelated business layer with the corresponding base mirror layer and dependency layer under the target architecture according to the mirror layer order saved in the mirror metadata, the mirror of the target architecture can be generated. The key to dynamic splicing is the retrieval of the corresponding base mirror layer and dependency layer under the target architecture. If the retrieval fails, splicing cannot be performed. Taking the tomcat:8.0.53 mirror of the X86 version as an example, its base mirror uses the openjdk:11 mirror of the X86 architecture released by the community. The process of generating the tomcat:8.0.53 mirror of the ARM architecture based on the tomcat:8.0.53 mirror of the X86 version is as follows:

[0093] Upload the openjdk:11 mirror of the X86 architecture to the mirror repository. Since the openjdk:11 is a mirror downloaded from the public network and belongs to a third-party mirror, no extended attribute information is saved in the mirror metadata. Therefore, when uploading, select the mirror type as the base mirror. The upload process triggers the mirror metadata repository update logic. According to the metadata information saved in the mirror, add the openjdk:11 mirror information to the mirror metadata repository. The mirror name is openjdk, the version is 11, and the architecture is X86. Also, add the mirror layer information and mirror layer relationship that make up the mirror; then add the information of each mirror layer, set the mirror layer type as the base environment layer, and add information such as the mirror layer fingerprint;

[0094] Upload the openjdk:11 mirror of the ARM architecture to the mirror repository. The upload process is the same as that of the X86 architecture mirror;

[0095] Create the tomcat:8.0.53 mirror of the X86 version. Select openjdk:11 as the base mirror. Since the target mirror architecture is X86, the X86 version of the openjdk:11 mirror will be selected as the base mirror, and run the mirror layer trimmer. Since all mirror layers of openjdk:11 are marked as base mirror layers, they are all trimmed. At the same time, update the metadata file json of the trimmed mirror layer in the tomcat:8.0.53 mirror, and add a trim mark in the Labels field; then run the mirror layer merger to merge the remaining layers into the business layer; finally, run the mirror layer scanner to scan the business layer and mark its type as architecture-unrelated, update the metadata file json of the business layer, add a mirror layer type mark in the Labels field, package and output the mirror, and update the mirror metadata repository at the same time;

[0096] Based on the X86 version of the tomcat:8.0.53 image, an ARM version of the tomcat:8.0.53 image is built. First, the type of the business layer is detected. The business layer of the tomcat:8.0.53 image is architecture-independent and meets the splicing requirements. Then, the remaining image layers are scanned. In this example, except for the business layer, other image layers are all common image layers and are trimmed. And all the common image layers come from the third-party image openjdk:11. Since the third-party image openjdk:11 does not use extended attributes, this step cannot retrieve the image layer with the same function tags under the ARM architecture through the function tags of the trimmed image layer. Here, it is necessary to find the corresponding base image based on the trimmed image layer, and then find the base image with the same name under the ARM architecture according to the base image name. Finally, the base image under the ARM architecture and the business layer are spliced to form the target image; in this example, first find the X86 architecture openjdk:11 base image according to the trimmed image layer information, then find the base image under the ARM architecture according to the image identifier openjdk:11, and finally splice all the image layers of the business layer and the openjdk:11 under the ARM architecture into the tomcat:8.0.53 image under the ARM architecture.

[0097] The difference between semi-architecture-related applications and architecture-independent applications lies in that semi-architecture-related applications contain some architecture-strongly-related files. When generating the target architecture image, it is necessary to provide the corresponding files under the target architecture. To reduce the system complexity, this solution limits the number of architecture-strongly-related files. Considering that in real-world scenarios, semi-architecture-related applications usually only contain a small number of architecture-strongly-related files, the limit value is set to 20. When the number of architecture-strongly-related files exceeds 20, the image layer will be converted to the architecture-strongly-related type, and the corresponding application type will be converted to the architecture-strongly-related application. The reason for this is that if the image layer contains too many architecture-strongly-related files, the benefits brought by dynamic splicing technology will be reduced, and the system will be more complex; to facilitate the quick matching of architecture-strongly-related files under the target architecture and simplify the system, this solution makes the following regulations:

[0098] For semi-architecture-related applications, the total number of architecture-strongly-related files they carry cannot exceed 20 (the specific implementation can be adjusted according to needs);

[0099] When making the image, files of a specific architecture must be copied to the / opt / app / dep / directory, and the directory name is named after the architecture name, following industry specifications. For example, the directory name for storing X86 64-bit architecture-related files is amd64, and the directory name for storing arm64-bit architecture-related files is aarch64. The / opt / app / dep / arch directory is a symbolic link pointing to the specific architecture, linked to the target architecture directory to shield the architecture directory differences during image operation;

[0100] Architecture strongly related files can be written when making the image, or can be written using tools as needed. Usually, only the architecture related files of common architectures are written when making the image. For example, when making an X86 image, the X86 and ARM architecture related files are written. In this way, an ARM architecture image can be quickly generated based on the X86 architecture image. When other architectures such as Shenwei need to be supported later, the Shenwei architecture related binaries can be written through tools; before building the target architecture image using the method provided by this solution, the original image needs to contain the target architecture related binaries.

[0101] Still taking the tomcat:8.0.53 image as an example, assuming that the binary operation and maintenance tool devtool is included inside the image. When making the X86 version image, provide the X86 architecture and ARM architecture binary devtool. After the X86 version image is made, the view of the directory / opt / app / dep is as Figure 4 shown; for the generated ARM version image, the view of the directory / opt / app / dep is as Figure 5 shown.

[0102] Architecture related files can also be added later through tools. Taking the example of adding Shenwei architecture support to the X86 version tomcat:8.0.53 image, run the image layer updater, provide the Shenwei version binary devtool, and add the Shenwei architecture to the X86 version. Subsequently, a Shenwei architecture image can be quickly generated based on the X86 version image, and the view of the directory / opt / app / dep is as Figure 6 shown.

[0103] For applications strongly related to the architecture, since the binaries are related to specific CPU architectures, it is impossible to generate the image of the target architecture through dynamic splicing.

[0104] After the trimmed and merged image is distributed to the target system and uploaded to the target system image repository, query the target system image metadata repository to determine whether the trimmed image layer exists. If the trimmed image layer does not exist, this upload fails.

[0105] The architecture relevance of the image is shielded by the node-agent running on the node. The node-agent adopts different processing flows according to the different types of the business layer of the image to be started, which are specifically as follows:

[0106] Architecture - independent: If the business layer exists in the image to be started and the business layer type recorded in the image metadata repository is architecture - independent, it indicates that the image can be dynamically spliced to generate an image of the target architecture (since the business layer is architecture - independent, it can be spliced with the common image layer of any architecture). The node - agent queries whether the current image in the image metadata repository has been trimmed. If so, it determines whether there are missing image layers in the local repository. The matching logic first retrieves the image layer with the same function label under the target architecture according to the function label of the image layer. If the function label retrieval fails, it retrieves the base image containing the trimmed image layer according to the missing image layer, retrieves the image with the same name under the target architecture according to the base image name, and uses the image layer of the image with the same name under the target architecture as the missing image layer. It pulls the trimmed image layer of the same architecture as the current node and splices them one by one according to the order of the image layers recorded in the image metadata repository, with the business layer spliced to the top layer. After splicing is completed, the image is started.

[0107] Highly architecture - dependent: No dynamic splicing is performed. If there is no image with the same name and version under the target architecture in the image repository, the image startup fails.

[0108] Semi - architecture - dependent: The node - agent queries the image metadata repository to check whether the current image has been trimmed. If so, it queries the image metadata repository, retrieves and pulls the trimmed image layer of the same architecture as the current node. The specific process is the same as that of architecture - related applications. The difference is that the highly architecture - related files included in semi - architecture - related applications need special processing. The node - agent queries whether there is a sub - directory with the same name as the target architecture in the business layer directory / opt / data / dep. If so, it creates a symbolic link / opt / data / dep / arch to point to the target architecture directory. Finally, the image is spliced and started.

[0109] Missing business layer: No dynamic splicing is performed. If the target image exists, it is pulled and started; otherwise, the image startup fails.

[0110] As a further improvement of this solution, in S5, under the existing technology system, image pulling starts only after the node receives a container startup request, which greatly reduces the system response speed and cannot support the operation of high - performance systems, such as the rapid online of WEB services and the rapid elastic scaling requirements of AI modules.

[0111] The image pre - heating technology advances the image pulling time and reduces the image startup time to obtain a good response speed. The difficulty of image pre - heating lies in how to accurately predict the image startup time. Combining with the layer - based image management technology introduced in S1 of the present invention, the following several solutions can be adopted for image pre - heating:

[0112] Periodic preheating: Periodic preheating is used to preheat the public image layer, which can be referenced by multiple images. The node-agent running on the local node regularly interacts with the image metadata repository to obtain public image layer change information and generate image layer pull tasks. When the node load drops to the preset threshold, the pull task is started to update the local image layer.

[0113] Trigger preheating: Image preheating is triggered by changes in node labels. Containers are usually scheduled based on labels. A new label indicates that a new container will be started. When the node-agent running on the node detects a change in the current node label, the node-agent matches images that can run on the current node based on the latest label. Images that can run on the current node but whose image layers have not yet been pulled are added to the queue to be pulled. The image pulling module pulls them locally when the load is idle.

[0114] Predictive preheating: Triggered preheating is suitable for scenarios where tags and images are strongly bound together. The images running on nodes can be determined based on tag filtering. However, some cloud platform applications don't rely solely on tags for their available nodes. Instead, the node on which a container is started is determined based on the current application load and available node resources. These applications are typically highly scalable, such as AI models or web services. They typically maintain a small number of containers to provide services, requiring rapid scaling when application load increases and scaling down when load decreases. Using triggered preheating can waste resources. After pulling an image, the application load may remain low, eliminating the need for scaling. Or, when scaling is required, the current node's available resources may not be sufficient to run the container. Compared to triggered preheating, predictive preheating can more accurately predict the container's running node. The prediction module running on the management node comprehensively considers application scaling requirements and node load. When the application load exceeds the warning value (the warning value is below the elastic scaling threshold), the prediction module runs a preselection algorithm based on cluster node load and application scaling configuration, selects the target node, and sends the image pull task to the node-agent service on the target node, which completes the image pull.

[0115] The image recycling strategy affects the image preheating effect. Images pulled to the local server are cleaned up due to storage shortages or because they have not been used for a long time and trigger timeout cleaning. Due to the dynamic nature of distributed systems, the cleaned images may be started on the same node later, resulting in resource waste.

[0116] Adopting a layer-based priority recycling strategy can reduce the impact on image preheating. By increasing the priority of public image layers, delaying the recycling of public image layers, and retaining public image layers in the node local warehouse as much as possible, it helps improve overall resource utilization.

[0117] Compared with the prior art, the present invention has the following beneficial effects:

[0118] 1. By introducing layer-based image management technology, the present invention adjusts the image management granularity from the image unit to the image layer unit. The image layer stores specific content and uses a unique fingerprint identifier. Images can be separated and spliced on a per-layer basis as needed, providing technical support for image tailoring, rapid generation of heterogeneous images, and image preheating. The layer-based image management technology stores image layer information and the relationships between image layers in the image metadata repository, making zero intrusion into the existing image specifications and being compatible with existing images, thereby solving problems such as large occupancy of image storage space, low image distribution efficiency, slow image startup speed, the need to remanufacture images when adapting to new architectures, and high maintenance costs for multi-architecture images in the cloud-native environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 is the in-layer tailoring diagram of the present invention;

[0120] Figure 2 is the flowchart of public image layer tailoring of the present invention;

[0121] Figure 3 is the schematic diagram of public image layer tailoring of the present invention;

[0122] Figure 4 is the X86 architecture directory view of the present invention;

[0123] Figure 5 is the ARM architecture directory view of the present invention;

[0124] Figure 6 is the X86 architecture directory view of the present invention after adding support for the ShenWei architecture;

[0125] Figure 7 is the schematic diagram of the content of the Dockerfile of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0126] The following further describes in detail the embodiments of the present invention in conjunction with the drawings and embodiments. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0127] In this embodiment, as Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7As shown in the figure, a method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform. By introducing layer-based mirror image management technology, the mirror image management granularity is adjusted from the mirror image unit to the mirror image layer unit. The mirror image layer stores specific content and uses a unique fingerprint identifier. The mirror image can be separated and spliced on a per-layer basis as needed, providing technical support for mirror image cropping, rapid generation of heterogeneous mirror images, and mirror image preheating. The layer-based mirror image management technology stores the mirror image layer information and the relationships between mirror image layers in the mirror image metadata warehouse, without any intrusion into the existing mirror image specifications and being compatible with existing mirror images. The method for mirror image cropping and multi-architecture mirror image construction includes the following steps:

[0128] Taking the Tomcat application as an example, this solution describes the implementation process in the order of mirror image production, in-mirror-image-layer cropping, common mirror image layer cropping, mirror image upload, mirror image preheating, and mirror image startup. This example requires a total of 2 sets of clusters: Cluster A and Cluster B. The information of Cluster A is as follows:

[0129]

[0130] Table 1 Information of Cluster A The information of Cluster B is as follows:

[0131]

[0132] Table 2 Information of Cluster B The list of materials required for this example is as follows:

[0133]

[0134]

[0135] Table 3 List of materials

[0136] Mirror image production: Mirror image production is carried out on the master node of Cluster A, and the process is as follows:

[0137] 1. Upload the openjdk:11 mirror image of the X86 architecture to the mirror image repositories of Cluster A and Cluster B. When uploading, select the mirror image type as the base mirror image. The upload process triggers the mirror image metadata warehouse update logic, adds the openjdk:11 information to the mirror image metadata warehouse, and marks all mirror image layer types as base mirror image layers;

[0138] 2. Upload the openjdk:11 mirror image of the ARM architecture to the mirror image repositories of Cluster A and Cluster B. The upload process is the same as that of the X86 architecture mirror image;

[0139] 3. Produce the tomcat:8.0.53 mirror image of the X86 version on the master node of Cluster A, and select openjdk:11 as the base mirror image;

[0140] The key content of the Dockerfile used for mirror image production is as Figure 7as shown

[0141] Mirror layer clipping and common mirror layer clipping: Run the mirror layer clipper to scan all mirror layers in sequence, delete the files under the / tmp directory in the mirror layer, and if the current mirror layer type is a common mirror layer, clip it and update the mirror layer metadata;

[0142] Run the mirror layer merger to merge the remaining mirror layers of tomcat:8.0.53 into the business layer;

[0143] Run the mirror layer scanner to scan the business layer of the tomcat:8.0.53 mirror and mark its type as architecture - irrelevant, update the business layer metadata file json, add a mirror layer type marker in the Labels field, package and output the mirror tomcat_8.0.53 - amd64.tar, and at the same time update the mirror metadata repository.

[0144] Mirror upload: Upload the tomcat_8.0.53 - amd64.tar mirror generated in the previous step to the mirror repository of Cluster B. When uploading, all mirror layers will be scanned. If the current mirror layer has been clipped, query whether the clipped mirror layer exists in the mirror metadata repository. If it does not exist, report an error and exit;

[0145] Parse all mirror layers in sequence, and update the mirror layer information to the mirror metadata repository according to the information in the Labels field of the mirror layer json file.

[0146] Mirror warm - up: This example demonstrates periodic warm - up, taking nodes node5 and node6 of Cluster B as examples.

[0147] When uploading the base image openjdk:11 to Cluster B, the node - agent running on the node detects the base mirror layer update event, generates a mirror layer pull task, and pulls the base mirror layer to the current node. Node5 pulls the ARM - architecture base mirror layer, and node6 pulls the X86 - architecture base mirror layer.

[0148] Mirror startup: Create a K8S StatefulSet on the master node of Cluster B to start 3 mutually exclusive Tomcat containers (mutually exclusive means that no two or more Tomcat containers can run on the same node). The mirror used is tomcat:8.0.53. This example does not show the content of the StatefulSet. Readers who need it can refer to the K8S specification. Taking the node5 node of Cluster B as an example, the startup process is as follows:

[0149] The node-agent on the node5 node obtains the to-be-started image from the image metadata repository. The type of the image business layer is architecture-independent. It obtains the list of trimmed common image layers from the image metadata repository and queries whether the trimmed common image layers exist in the local repository of the current node. Since the common image layers of the ARM architecture have been pulled to the local during image preheating, the pulling of image layers will be skipped. Then, it runs the image splicer to splice all the common image layers of the ARM architecture base image openjdk:11 with the tomcat:8.0.53 business layer (since all the image layers exist in the local repository of node5, only the metadata needs to be updated during splicing). Finally, the container is started.

[0150] The embodiments of the present invention are given for the purpose of illustration and description, and are not exhaustive or limit the present invention to the disclosed form. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features.

Claims

1. A method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform, characterized in that, The method for mirror cropping and multi-architecture mirror building includes the following steps: S1: Introduce layer-based management technology, and divide the mirror layers into a common mirror layer for storing the basic running environment, a file copy layer for storing temporary files, and a business mirror layer for storing applications; S2: Mirror layer cropping. When generating the mirror layer, identify and recognize the temporary data in the mirror layer and delete it when generating the mirror, so as to reduce the size of a single-layer mirror; S3: Common mirror layer cropping. Identify and recognize the common mirror layer in the mirror, and delete the common mirror layer to reduce the mirror size; S4: Rapid generation of heterogeneous mirrors. Scan and identify the business mirror layer. When adapting to a new architecture, quickly generate a mirror of another architecture based on the properties of the business mirror layer for one architecture; In S4, the rapid generation of heterogeneous mirrors is based on the mirror layer dynamic splicing technology. The mirror layer dynamic splicing technology is applicable to architecture-unrelated and semi-architecture-related applications. For architecture-unrelated applications, their architecture relevance is shielded by the basic mirror layer and the dependency layer. By merging the architecture-unrelated business layer with the corresponding basic mirror layer and dependency layer under the target architecture according to the mirror layer order saved in the mirror metadata, the mirror of the target architecture can be generated; Semi-architecture-related applications contain some architecture-strongly-related files. When generating the mirror of the target architecture, it is necessary to provide the corresponding files under the target architecture to reduce the system complexity; Among them, architecture-unrelated: The application does not need to perceive the CPU architecture; Semi-architecture-related: The application main body is developed by architecture-unrelated languages and carries a small number of binary libraries of architecture-related languages; Architecture-strongly-related: The application is bound to the CPU architecture and can only run on the target architecture CPU; S5: Mirror preheating. Move the mirror pulling time forward to reduce the mirror startup time and obtain a good response speed.

2. The method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform according to claim 1, wherein: In S1, the layer-based mirror management technology divides the mirror layer into a basic environment layer, a dependency layer, a file copy layer, and a business layer according to functions: Basic mirror layer: Store the basic running environment of the OS and the architecture; Dependency layer: Store the direct dependencies during application runtime, including the runtime environment and dynamic libraries; Common mirror layer: The basic mirror layer and the dependency layer are collectively referred to as the common mirror layer. The common mirror layer belongs to common resources and can be shared by multiple mirrors; File copy layer: The content stored in this layer is temporary and plays an auxiliary role during mirror production. After the mirror production is completed, the data in this layer is no longer needed; Business layer: Store the data and configurations of the application itself.

3. The method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform according to claim 1, wherein: In S1, introduce a mirror metadata warehouse to support layer-based mirror management technology. The mirror metadata warehouse stores mirror metadata information, including mirror information and mirror layer information, which describe the composition and attributes of the mirror.

4. The method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform according to claim 1, characterized in that: In S2, the in-mirror layer trimming is used to trim off the file copy layer, save the disposable data and temporary data used during mirror production in the / tmp directory. After the mirror production is completed, run the mirror layer trimmer to delete the data in the / tmp directory and update the mirror metadata information. The mirror contains multiple file copy layers, and the mirror layer trimmer processes all mirror layers in turn to empty the / tmp directory of the mirror layer.

5. The method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform according to claim 1, wherein: In S3, the common image layer includes a base image layer and a dependency layer. The clipping of the common image layer is completed by an image layer clipper. The image layer clipper extracts the metadata information of the image to be clipped, compares the fingerprint of each image layer that makes up the image with the fingerprint stored in the image metadata repository. If the image layer exists in the image metadata repository and the image layer type is a base image layer or a dependency layer, it will be clipped; otherwise, the current image layer will be skipped.

6. The method for mirror image cropping and multi-architecture mirror image construction based on a cloud platform according to claim 3, characterized in that: The clipping of the common image layer depends on the image metadata repository. Only when the image layer type stored in the image metadata repository is a common image layer can the corresponding image layer be clipped. There are two ways to update the image metadata repository: Automatic update: The image metadata repository is updated when uploading an image. If the uploaded image is a base image, all image layers are marked as base image layers; otherwise, the image layer scanner is run to dynamically identify the image layer type according to the built-in rules. The working principle of the image scanner will be described in the next subsection. Manual update: The administrator manually writes the image layer information into the image metadata repository.

Images