Mirror image file processing method and device

Abstract

The invention relates to a mirror image file processing method and device, and relates to the technical field of container mirror image generation. The method comprises the steps of obtaining target file system directory information of a target container in response to a mirror image file export request for the target container; the target file system directory information of the target container is used for representing file system directory information of a writable layer of the target container; comparing the target file system directory information of the basic mirror image with the target file system directory information of the target container to obtain a difference file of the target container relative to the basic mirror image; constructing a layer file of a new incremental layer based on the difference file; writing a layer file of the incremental layer into a mirror image layer file directory of the basic mirror image, and updating a list file and an index file of the basic mirror image to obtain a mirror image file of the target container; the manifest file comprises metadata of each mirror image layer in the basic mirror image, and the index file comprises metadata of the manifest file. By adopting the method, the export efficiency of the container mirror image file can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of container image generation, and in particular to a method and device for processing image files. Background Art

[0002] In artificial intelligence application scenarios, developers often need to export the running container environment as an image file so that they can solidify the debugging results and move them to the production environment.

[0003] In traditional technology, when exporting a container image file, the entire container image file is generally exported in its entirety, that is, a full copy is made; however, when the container size is large, the export speed of the container image file is slow, resulting in low export efficiency of the container image file.

[0004] Therefore, there is a technical problem in traditional technologies that the export efficiency of container image files is low. Summary of the Invention

[0005] The present application provides an image file processing method and device to solve the technical problem of low export efficiency of container image files.

[0006] In a first aspect, some embodiments provide a method for processing an image file, the method comprising:

[0007] In response to an image file export request for a target container, obtaining target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container;

[0008] Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file;

[0009] Building a new incremental layer file based on the difference file;

[0010] The layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

[0011] Technical effect: In response to a request to export an image file for a target container, the target file system directory information of the target container is obtained; the target file system directory information of the target container is used to represent the file system directory information of the writable layer of the target container; then the target file system directory information of the base image is compared with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file; then based on the difference file, a layer file of a new incremental layer is constructed; finally, the layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container; the manifest file includes metadata of each image layer in the base image, and the index file includes metadata of the manifest file. In this way, when exporting a container image file, it is only necessary to build a new incremental layer file based on the difference file of the target container relative to the base image, and write the incremental layer layer file into the image layer file directory of the base image, as well as update the manifest file and index file of the base image. There is no need to export the entire image file of the target container, that is, there is no need to copy the entire file, which simplifies the container image file export process, helps to improve the export speed of the container image file, thereby improving the export efficiency of the container image file, and avoiding the defect that the export speed of the container image file is slow, resulting in low export efficiency of the container image file.

[0012] In some embodiments of the present application, before obtaining target file system directory information of the target container in response to an image file export request for the target container, the method further includes:

[0013] Download the base image from the image repository;

[0014] Generate target file system directory information of the base image according to a preset image layout format.

[0015] Technical effect: First, the base image is downloaded from the image repository, and then the target file system directory information of the base image is generated according to the preset image layout format; in this way, by obtaining the target file system directory information of the base image in advance, it is convenient to subsequently compare the target file system directory information of the base image with the target file system directory information of the target container, and obtain the difference file of the target container relative to the base image, thereby improving the accuracy of determining the difference file.

[0016] In some embodiments of the present application, downloading the base image from the image repository includes:

[0017] Obtain the identification information of the image layer of the base image in the image repository, and obtain the identification information of the image layer stored in the local database;

[0018] Compare the identification information of the image layer of the base image in the image repository with the identification information of the image layer stored in the local database to obtain a comparison result;

[0019] Based on the comparison result, determining the image layer to be downloaded of the base image in the image repository; the image layer to be downloaded is used to represent the image layer contained in the base image in the image repository and not stored in the local database;

[0020] Download the image layer to be downloaded of the base image from the image warehouse.

[0021] Technical effect: first obtain the identification information of the image layer of the base image in the image warehouse, and obtain the identification information of the image layer stored in the local database, then compare the identification information of the image layer of the base image in the image warehouse with the identification information of the image layer stored in the local database to obtain a comparison result, and then determine the image layer to be downloaded of the base image in the image warehouse based on the comparison result, and finally download the image layer to be downloaded of the base image from the image warehouse; in this way, when downloading the base image from the image warehouse, by comparing the identification information of the image layer of the base image in the image warehouse with the identification information of the image layer stored in the local database, the image layer to be downloaded of the base image in the image warehouse is determined, and the image layer to be downloaded of the base image is downloaded, thereby achieving the purpose of only downloading the image layer contained in the base image in the image warehouse and not stored in the local database, which is beneficial to reducing file transfer redundancy, thereby improving the download speed of the base image, and further improving the download efficiency of the base image.

[0022] In some embodiments of the present application, in response to an image file export request for a target container, obtaining target file system directory information of the target container includes:

[0023] In response to an image file export request for a target container, obtaining a container identifier of the target container;

[0024] Obtaining the mount information file corresponding to the container identifier of the target container;

[0025] Identifying file system layering information of an overlay layer of the target container from the mount information file;

[0026] Identifying file system directory information of a writable layer of the target container from the file system layer information;

[0027] Based on the file system directory information of the writable layer of the target container, target file system directory information of the target container is obtained.

[0028] Technical effect: first, in response to an image file export request for a target container, the container identifier of the target container is obtained, then the mount information file corresponding to the container identifier of the target container is obtained, and the file system layering information of the overlay layer of the target container is identified from the mount information file, and then the file system directory information of the writable layer of the target container is identified from the file system layering information, and finally, based on the file system directory information of the writable layer of the target container, the target file system directory information of the target container is obtained; in this way, by identifying the file system directory information of the writable layer of the target container from the mount information file corresponding to the container identifier of the target container, and obtaining the target file system directory information of the target container based on the file system directory information of the writable layer of the target container, the purpose of determining the target file system directory information of the target container based on the file system directory information of the writable layer of the target container is achieved, which is conducive to improving the accuracy of determining the target file system directory information of the target container.

[0029] In some embodiments of the present application, comparing the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image includes:

[0030] Decompressing the target file system directory information of the base image into the first temporary directory information, and mounting the target file system directory information of the target container into the second temporary directory information; the directory structure of the first temporary directory information and the second temporary directory information is the same;

[0031] The first temporary directory information and the second temporary directory information are compared to obtain the difference file.

[0032] Technical effect: first, the target file system directory information of the base image is decompressed to the first temporary directory information, and the target file system directory information of the target container is mounted to the second temporary directory information, and then the first temporary directory information and the second temporary directory information are compared to obtain a difference file; in this way, when determining the difference file, the target file system directory information of the base image is decompressed to the first temporary directory information, and the target file system directory information of the target container is mounted to the second temporary directory information, and the directory structure of the first temporary directory information and the second temporary directory information is the same, which is conducive to accurately obtaining the difference file by comparing the first temporary directory information and the second temporary directory information, thereby improving the accuracy of determining the difference file.

[0033] In some embodiments of the present application, the comparing the first temporary directory information and the second temporary directory information to obtain the difference file includes:

[0034] identifying a file that exists in the second temporary directory information and does not exist in the first temporary directory information as a new file in the target container;

[0035] identifying a file having the same file path and different file content in the second temporary directory information and the first temporary directory information as a modified file of the target container;

[0036] identifying a file that exists in the first temporary directory information and does not exist in the second temporary directory information as a deleted file of the target container;

[0037] The difference file is obtained based on the newly added file, the modified file and the deleted file.

[0038] Technical effect: By identifying files that exist in the second temporary directory information but do not exist in the first temporary directory information as newly added files of the target container, and identifying files with the same file path but different file content in the second temporary directory information and the first temporary directory information as modified files of the target container, and identifying files that exist in the first temporary directory information but do not exist in the second temporary directory information as deleted files of the target container, finally, based on the newly added files, modified files and deleted files, a difference file is obtained; in this way, the newly added files, modified files and deleted files of the target container are determined in different ways, and based on the newly added files, modified files and deleted files of the target container, the difference file of the target container relative to the base image is obtained, which is conducive to improving the accuracy of determining the difference file.

[0039] In some embodiments of the present application, constructing a layer file of a new incremental layer based on the difference file includes:

[0040] Generate a deletion mark file corresponding to the deletion file;

[0041] Generate a layer file in a preset format based on the newly added file, the modified file, and the deletion mark file according to the same directory structure as the base image;

[0042] Based on the layer file in the preset format, a layer file of a new incremental layer is obtained.

[0043] Technical effect: By generating a deletion marker file corresponding to the deleted file, and then generating a layer file in a preset format based on the newly added files, modified files and deletion marker file according to the same directory structure as the basic image, and finally obtaining a new incremental layer layer file based on the preset format layer file; in this way, when exporting the container image file, only the new incremental layer layer file is generated, and there is no need to export the layer file of each image layer, thereby reducing file transfer redundancy and helping to improve the export efficiency of the container image file.

[0044] In some embodiments of the present application, writing the layer file of the incremental layer into the image layer file directory of the base image, and updating the manifest file and index file of the base image to obtain the image file of the target container includes:

[0045] Determine the file identifier of the layer file of the incremental layer;

[0046] According to the file identifier of the layer file of the incremental layer, the layer file of the incremental layer is written into the location corresponding to the file identifier in the image layer file directory of the base image.

[0047] Technical effect: By determining the file identifier of the layer file of the incremental layer, and then writing the layer file of the incremental layer to the location corresponding to the file identifier in the image layer file directory of the base image according to the file identifier of the layer file of the incremental layer; in this way, when exporting the container image file, only the layer file of the incremental layer is written to the location corresponding to the file identifier of the layer file of the incremental layer in the image layer file directory of the base image, and there is no need to export the layer file of each image layer, thereby reducing file transfer redundancy, which is beneficial to improving the export speed of the container image file, thereby further improving the export efficiency of the container image file.

[0048] In some embodiments of the present application, writing the layer file of the incremental layer into the image layer file directory of the base image, and updating the manifest file and index file of the base image to obtain the image file of the target container includes:

[0049] Obtain metadata of the layer file of the incremental layer;

[0050] Add the metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image to obtain an updated manifest file;

[0051] Obtaining metadata of the updated manifest file;

[0052] Updating the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain an updated index file;

[0053] The image file of the target container is obtained based on the mirror layer file directory of the layer file written into the incremental layer, the updated manifest file, and the updated index file.

[0054] Technical effect: by obtaining the metadata of the layer file of the incremental layer, and then adding the metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image, an updated manifest file is obtained, and then the metadata of the updated manifest file is obtained, and the metadata of the manifest file in the index file is updated to the metadata of the updated manifest file to obtain an updated index file, and finally the image file of the target container is obtained based on the image layer file directory, updated manifest file and updated index file of the layer file written to the incremental layer; in this way, the purpose of obtaining the image file of the target container based on the image layer file directory, updated manifest file and updated index file of the layer file written to the incremental layer is achieved, that is, only the layer file of the incremental layer is written, and the corresponding manifest file and index file are updated, without exporting the layer file of each image layer, and without exporting the entire container image file, thereby reducing file transfer redundancy, which is conducive to improving the export efficiency of the container image file, and avoiding the defect of exporting the entire container image file, resulting in low export efficiency of the container image file.

[0055] In a second aspect, some embodiments further provide an image file processing device, comprising: a communication module and a processor. The processor is coupled to the communication module and configured to:

[0056] In response to an image file export request for a target container, obtaining target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container;

[0057] Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file;

[0058] Building a new incremental layer file based on the difference file;

[0059] The layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

[0060] Technical effect: When exporting a container image file, it is only necessary to build a new incremental layer file based on the difference file of the target container relative to the base image, and write the incremental layer layer file into the image layer file directory of the base image, as well as update the manifest file and index file of the base image. There is no need to export the entire image file of the target container, that is, there is no need to copy the entire file, thereby simplifying the container image file export process, which is conducive to improving the export speed of the container image file, thereby improving the export efficiency of the container image file, and avoiding the defect that the export speed of the container image file is slow and the export efficiency of the container image file is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A diagram illustrating an application environment of the image file processing method provided in some embodiments of the present application;

[0063] Figure 2 A flowchart of an image file processing method provided in some embodiments of the present application;

[0064] Figure 3 A flowchart illustrating steps for generating target file system directory information of a base image provided in some embodiments of the present application;

[0065] Figure 4 A flowchart illustrating the steps of downloading a base image from an image repository provided in some embodiments of the present application;

[0066] Figure 5 A flowchart illustrating steps for obtaining target file system directory information of a target container provided in some embodiments of the present application;

[0067] Figure 6 A flowchart illustrating steps for obtaining a difference file of a target container relative to a base image provided in some embodiments of the present application;

[0068] Figure 7 A flowchart of steps for obtaining a difference file provided in some embodiments of the present application;

[0069] Figure 8 A flowchart illustrating steps for constructing a layer file for a new incremental layer provided in some embodiments of the present application;

[0070] Figure 9A flowchart illustrating steps for writing layer files of an incremental layer into an image layer file directory of a base image, provided in some embodiments of the present application;

[0071] Figure 10 A flowchart illustrating steps for obtaining an image file of a target container provided in some embodiments of the present application;

[0072] Figure 11 An architectural diagram of the container image layered storage and copy-on-write mechanism provided in some embodiments of the present application;

[0073] Figure 12 A schematic diagram of the file structure composition of a container image provided in some embodiments of the present application;

[0074] Figure 13 A schematic diagram of the layered architecture of OverlayFS (overlay file system) provided in some embodiments of the present application;

[0075] Figure 14 A signaling interaction diagram of the image file processing method provided in some embodiments of the present application;

[0076] Figure 15 FIG. 4 is a structural block diagram of an image file processing device in an embodiment. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0078] The image file processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. 1 , the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104 or placed on a cloud or other network server. Specifically, refer to FIG. Figure 1Terminal 102 sends an image file export request for the target container to server 104; server 104, in response to the image file export request for the target container, obtains target file system directory information for the target container; the target file system directory information for the target container is used to represent the file system directory information of the writable layer of the target container; server 104 compares the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file between the target container and the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file; server 104 constructs a new incremental layer layer file based on the difference file, writes the incremental layer layer file to the image layer file directory of the base image, and updates the manifest file and index file of the base image to obtain the image file of the target container; the manifest file includes metadata for each image layer in the base image, and the index file includes metadata for the manifest file. Terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, and tablet computers. Server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0079] In some embodiments, as Figure 2 As shown, a method for processing an image file is provided, which is applied to Figure 1 The server in the example is used to illustrate the method, which includes the following steps S201 to S204.

[0080] Step S201 : in response to an image file export request for a target container, obtain target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container.

[0081] Step S202 : Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file.

[0082] Step S203: construct a layer file of a new incremental layer based on the difference file.

[0083] Step S204, write the layer file of the incremental layer into the image layer file directory of the base image, and update the manifest file and index file of the base image to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

[0084] The target container refers to the container to which the image file needs to be exported, which can be a container selected by the user. It should be noted that the target container of this application refers to a large volume container in the AI ​​(Artificial Intelligence) scenario, such as more than 10GB.

[0085] The image file export request refers to a request to export the image file of the target container.

[0086] Among them, the target file system directory information of the target container includes the difference files generated when the target container is running on the basis of the base image, which is used to record the new files, modified files, and deleted files generated when the target container is running on the basis of the base image. Specifically, it refers to the file system directory information of the writable layer of the target container. The file system directory information of the writable layer of the target container includes the new files, modified files, and deleted files of the target container. For example, when the target container is running, all modifications (such as new files, modified files, and deleted files) will be written to the writable layer of the target container. That is, during the operation of the target container, the writable layer of the target container records all file system changes (new, modified, and deleted files).

[0087] For example, refer to Figure 11 The Ubuntu 22.04 image at the bottom represents the base operating system image, the cornerstone of the container file system and containing the most core system files. Layers 1 through 4 are read-only image layers, each storing a portion of the file system contents. The core of image layering design is reuse: multiple containers / images can share these read-only layers (avoiding duplicate storage). During runtime, each container (Container 1, Container 2, ..., Container n) appends a writable layer (a copy-on-write layer) on top of the read-only layer. Furthermore, when the container is running, if a file in the read-only layer is modified, it is first copied to the writable layer, and then the copy is modified (the original file in the read-only layer remains unchanged).

[0088] The base image refers to the image used as the baseline.

[0089] The target file system directory information of the base image includes the layer files of each image layer of the base image. For example, refer to Figure 12 ,The container image includes layer files, manifest files, configuration files, and auxiliary files of multiple image layers. The layer files of each image layer include layered compression packages (i.e., actual file contents), layered metadata (such as summary, size), and version identifiers.

[0090] Among them, differential files refer to files that are different from the base image, including newly added files, modified files, and deleted files.

[0091] The new incremental layer refers to the image layer that contains only differential files, which is stored in the form of a compressed tar package (tar.gz) and contains new files, modified files, and whiteout files (recording deletion operations).

[0092] Among them, the image layer file directory of the base image stores the layer files of each image layer of the base image, specifically blobs / sha256 / <digest>Table of contents.

[0093] The base image's manifest file, specifically manifest.json, contains metadata for each image layer within the base image. The metadata for each image layer includes its media type, size, and summary.

[0094] The index file of the base image includes metadata of multiple manifest files, specifically index.json; the metadata of the manifest files includes the size and summary of the manifest files.

[0095] Specifically, in response to the user's image file export operation for the target container, the terminal generates an image file export request for the target container, and sends the image file export request for the target container to the server. In response to an image file export request for a target container, the server obtains file system directory information of the writable layer of the target container, and uses the file system directory information of the writable layer of the target container as the target file system directory information of the target container, then obtains the target file system directory information of the base image, and compares the target file system directory information of the base image with the target file system directory information of the target container to obtain new files, modified files, and deleted files of the target container relative to the base image, and uses the new files, modified files, and deleted files of the target container relative to the base image as difference files of the target container relative to the base image; for example, the server identifies files that exist in the target file system directory information of the target container but do not exist in the target file system directory information of the base image as new files of the target container; identifies files that have the same file path in the target file system directory information of the target container and the target file system directory information of the base image but different file content as modified files of the target container; identifies files that exist in the target file system directory information of the base image but do not exist in the target file system directory information of the target container as deleted files of the target container; and uses the new files, modified files, and deleted files as difference files. Then, the server builds the corresponding image layer based on the difference file as a new incremental layer, and finally writes the layer file of the new incremental layer to the corresponding location in the image layer file directory of the base image, and updates the manifest file and index file of the base image to obtain the image file of the target container.

[0096] The technical solution provided by this embodiment obtains the target file system directory information of the target container in response to the image file export request for the target container; the target file system directory information of the target container is used to represent the file system directory information of the writable layer of the target container; then the target file system directory information of the base image is compared with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file; then based on the difference file, a layer file of a new incremental layer is constructed; finally, the layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container; the manifest file includes metadata of each image layer in the base image, and the index file includes metadata of the manifest file. In this way, when exporting a container image file, it is only necessary to build a new incremental layer file based on the difference file of the target container relative to the base image, and write the incremental layer layer file into the image layer file directory of the base image, as well as update the manifest file and index file of the base image. There is no need to export the entire image file of the target container, that is, there is no need to copy the entire file, which simplifies the container image file export process, helps to improve the export speed of the container image file, thereby improving the export efficiency of the container image file, and avoiding the defect that the export speed of the container image file is slow, resulting in low export efficiency of the container image file.

[0097] In some embodiments, as Figure 3 As shown, the above step 201, before obtaining the target file system directory information of the target container in response to the image file export request for the target container, also includes the step of generating the target file system directory information of the base image, specifically including the following steps S301 to S302. Among them:

[0098] Step S301: Download the base image from the image repository.

[0099] Step S302: Generate target file system directory information of the base image according to a preset image layout format.

[0100] Among them, the image warehouse refers to a database that stores multiple images.

[0101] The preset image layout format refers to the OCI (Open Container Initiative) image layout specification.

[0102] The target file system directory information of the base image includes directory structures such as oci-layout, index.json, and blobs / sha256. oci-layout is used to mark this as an image storage layout that complies with the OCI specification and records basic metadata such as the layout version. index.json refers to the image index file, which is used to associate the manifest (image manifest, which describes the layers, configuration, and other information contained in the image) of different platform variants. blobs / sha256 is used to store the data of each layer of the image (files are named after the sha256 hash value) and related metadata, and is where the image content is actually stored.

[0103] Specifically, the server downloads the corresponding base image from the image repository through the image download instruction, and then generates the file system directory information of the base image that meets the preset image layout format according to the preset image layout format, and finally uses the file system directory information of the base image that meets the preset image layout format as the target file system directory information of the base image.

[0104] The technical solution provided by this embodiment first downloads the base image from the image repository, and then generates the target file system directory information of the base image according to a preset image layout format; in this way, by obtaining the target file system directory information of the base image in advance, it is convenient to subsequently compare the target file system directory information of the base image with the target file system directory information of the target container to obtain the difference file of the target container relative to the base image, thereby improving the accuracy of determining the difference file.

[0105] In some embodiments, as Figure 4 As shown, the above step S301, downloading the base image from the image repository, specifically includes the following steps S401 to S404. Among them:

[0106] Step S401: Obtain identification information of the image layer of the base image in the image repository, and obtain identification information of the image layer stored in the local database.

[0107] Step S402 : Compare the identification information of the image layer of the base image in the image repository with the identification information of the image layer stored in the local database to obtain a comparison result.

[0108] Step S403: Based on the comparison result, determine the image layer to be downloaded of the base image in the image repository; the image layer to be downloaded is used to represent the image layer included in the base image in the image repository and not stored in the local database.

[0109] Step S404: Download the image layer to be downloaded of the base image from the image repository.

[0110] The identification information of the image layer may refer to a hash value of the image layer, such as a sha256 hash value.

[0111] The image layer to be downloaded of the base image refers to the image layer contained in the base image in the image repository and not stored in the local database, specifically the image layer that needs to be downloaded in the base image.

[0112] Specifically, the server calculates the hash value of the image layer of the base image in the image warehouse through a hash algorithm (such as the sha256 algorithm), and uses the hash value of the image layer of the base image in the image warehouse as the identification information of the image layer of the base image in the image warehouse, and then calculates the hash value of the image layer stored in the local database through a hash algorithm (such as the sha256 algorithm), and uses the hash value of the image layer stored in the local database as the identification information of the image layer stored in the local database; then the server compares the identification information of the image layer of the base image in the image warehouse with the identification information of the image layer stored in the local database to obtain a comparison result; for example, if the base image in the image warehouse If the identification information of the image layer of the base image is the same as the identification information of the image layer stored in the local database, that is, the image layer of the base image has been stored in the local database and does not need to be downloaded. If the identification information of the image layer of the base image in the image warehouse is different from the identification information of the image layer stored in the local database, that is, the image layer of the base image is not stored in the local database and needs to be downloaded; then, based on the comparison result, the image layers contained in the base image in the image warehouse and not stored in the local database are determined, and these image layers are used as the image layers to be downloaded of the base image in the image warehouse. Finally, the image layers to be downloaded of the base image are downloaded from the image warehouse through the image download instruction.

[0113] The technical solution provided by this embodiment first obtains the identification information of the image layer of the base image in the image warehouse and the identification information of the image layer stored in the local database, then compares the identification information of the image layer of the base image in the image warehouse with the identification information of the image layer stored in the local database to obtain a comparison result, and then determines the image layer to be downloaded of the base image in the image warehouse based on the comparison result, and finally downloads the image layer to be downloaded of the base image from the image warehouse; in this way, when downloading the base image from the image warehouse, by comparing the identification information of the image layer of the base image in the image warehouse with the identification information of the image layer stored in the local database, the image layer to be downloaded of the base image in the image warehouse is determined, and the image layer to be downloaded of the base image is downloaded, thereby achieving the purpose of downloading only the image layers contained in the base image in the image warehouse and not stored in the local database, which is beneficial to reducing file transmission redundancy, thereby improving the download speed of the base image, and further improving the download efficiency of the base image.

[0114] In some embodiments, as Figure 5 As shown, the above step S201, in response to the image file export request for the target container, obtains the target file system directory information of the target container, specifically including the following steps S501 to S505.

[0115] Step S501 : In response to an image file export request for a target container, a container identifier of the target container is obtained.

[0116] Step S502: Obtain the mount information file corresponding to the container identifier of the target container.

[0117] Step S503: Identify the file system layering information of the overlay layer of the target container from the mount information file.

[0118] Step S504: Identify the file system directory information of the writable layer of the target container from the file system layer information.

[0119] Step S505 : obtaining target file system directory information of the target container based on the file system directory information of the writable layer of the target container.

[0120] The container identifier of the target container refers to unique identification information of the target container, such as PID (Process ID).

[0121] The mount information file is used to record the container's file system mount details, including the file system layering information of the target container's overlay layer. In actual scenarios, the mount information file is read by reading / proc / <pid> / mountinfo is obtained.

[0122] The target container's overlay is as follows: Figure 13 The Overlay layer (merged layer / view layer) shown is the superposition result of the Lower layer (lower layer / read-only layer) and the Upper layer (upper layer / writable layer).

[0123] The target container's overlay file system layer information refers to the file system layer information of the Overlay layer, such as Overlay type entries. Overlay type entries refer to the file system layer information mounted via the Overlayfs storage driver. These entries record the key paths and parameters of the Overlayfs layer structure and are the core basis for parsing the container root file system mount status.

[0124] The file system layer information includes file system directory information of a writable layer of the target container and file system directory information of a read-only layer of the target container.

[0125] Specifically, the server responds to the image file export request for the target container, obtains the container identifier of the target container through the container identifier acquisition instruction, then uses the container identifier of the target container to obtain the corresponding mount information file, and uses the file system layering information identification instruction to identify the file system layering information of the target container's overlay layer (such as the Overlay layer), such as the Overlay type entry, from the mount information file; then uses the file system directory information identification instruction of the writable layer to identify the file system directory information of the writable layer of the target container from the file system layering information, and finally uses the file system directory information of the writable layer of the target container as the target file system directory information of the target container.

[0126] The technical solution provided by this embodiment first obtains the container identifier of the target container in response to an image file export request for the target container, then obtains the mount information file corresponding to the container identifier of the target container, and identifies the file system layering information of the overlay layer of the target container from the mount information file. Then, the file system directory information of the writable layer of the target container is identified from the file system layering information. Finally, the target file system directory information of the target container is obtained based on the file system directory information of the writable layer of the target container. In this way, by identifying the file system directory information of the writable layer of the target container from the mount information file corresponding to the container identifier of the target container and obtaining the target file system directory information of the target container based on the file system directory information of the writable layer of the target container, the purpose of determining the target file system directory information of the target container based on the file system directory information of the writable layer of the target container is achieved, which is conducive to improving the accuracy of determining the target file system directory information of the target container.

[0127] In some embodiments, as Figure 6 As shown, the above step S202 compares the target file system directory information of the base image with the target file system directory information of the target container to obtain the difference file of the target container relative to the base image, which specifically includes the following steps S601 to S602. Among them:

[0128] Step S601: decompress the target file system directory information of the base image into the first temporary directory information, and mount the target file system directory information of the target container into the second temporary directory information; the directory structure of the first temporary directory information and the second temporary directory information is the same.

[0129] Step S602: Compare the first temporary directory information with the second temporary directory information to obtain a difference file.

[0130] Among them, the first temporary directory information refers to the temporary directory information to which the target file system directory information of the base image is decompressed, such as / mnt / image-base; the second temporary directory information refers to another temporary directory information to which the target file system directory information of the target container is mounted, such as / mnt / overlay-upper.

[0131] It should be noted that the directory structures of the first temporary directory information and the second temporary directory information are the same in order to facilitate subsequent file comparison operations.

[0132] Specifically, the server uses the decompression instruction to decompress the target file system directory information of the base image to the corresponding temporary directory information, such as / mnt / image-base, and uses the temporary directory information as the first temporary directory information; then uses the mount instruction to mount the target file system directory information of the target container to another temporary directory information, such as / mnt / overlay-upper, and uses the other temporary directory information as the second temporary directory information; finally, through the file comparison instruction, the first temporary directory information and the second temporary directory information are compared to obtain the difference file of the target container relative to the base image.

[0133] The technical solution provided by this embodiment first decompresses the target file system directory information of the base image to the first temporary directory information, and mounts the target file system directory information of the target container to the second temporary directory information, and then compares the first temporary directory information and the second temporary directory information to obtain a difference file; in this way, when determining the difference file, the target file system directory information of the base image is decompressed to the first temporary directory information, and the target file system directory information of the target container is mounted to the second temporary directory information, and the directory structure of the first temporary directory information and the second temporary directory information is the same, which is conducive to accurately obtaining the difference file by comparing the first temporary directory information and the second temporary directory information, thereby improving the accuracy of determining the difference file.

[0134] In some embodiments, as Figure 7 As shown, the above step S602 compares the first temporary directory information with the second temporary directory information to obtain a difference file, which specifically includes the following steps S701 to S704. Among them:

[0135] Step S701: Identify a file that exists in the second temporary directory information but does not exist in the first temporary directory information as a new file in the target container.

[0136] Step S702: Identify files with the same file path and different file content in the second temporary directory information and the first temporary directory information as modified files of the target container.

[0137] Step S703: Identify files that exist in the first temporary directory information but do not exist in the second temporary directory information as deleted files of the target container.

[0138] Step S704: obtaining a difference file based on the newly added files, modified files, and deleted files.

[0139] The newly added files in the target container refer to files that exist in the second temporary directory information but do not exist in the first temporary directory information.

[0140] The modified file of the target container refers to a file having the same file path in the second temporary directory information and the same file content in the first temporary directory information.

[0141] The deleted files of the target container refer to files that exist in the first temporary directory information but do not exist in the second temporary directory information.

[0142] Specifically, the server uses file comparison instructions, such as diff, rsync, etc., to compare the first temporary directory information and the second temporary directory information to obtain files that exist in the second temporary directory information but do not exist in the first temporary directory information, files with the same file path in the second temporary directory information and the first temporary directory information but different file content, and files that exist in the first temporary directory information but do not exist in the second temporary directory information; then, the files that exist in the second temporary directory information and do not exist in the first temporary directory information are used as new files of the target container; the files with the same file path in the second temporary directory information and the first temporary directory information but different file content are used as modified files of the target container; the files that exist in the first temporary directory information but do not exist in the second temporary directory information are used as deleted files of the target container; finally, the new files, modified files, and deleted files are all used as difference files of the target container relative to the base image.

[0143] The technical solution provided by this embodiment identifies files that exist in the second temporary directory information but do not exist in the first temporary directory information as newly added files of the target container, identifies files with the same file path but different file content in the second temporary directory information and the first temporary directory information as modified files of the target container, and identifies files that exist in the first temporary directory information but do not exist in the second temporary directory information as deleted files of the target container. Finally, a difference file is obtained based on the newly added files, modified files, and deleted files. In this way, the newly added files, modified files, and deleted files of the target container are determined in different ways, and the difference file of the target container relative to the base image is obtained based on the newly added files, modified files, and deleted files of the target container, which is conducive to improving the accuracy of determining the difference file.

[0144] In some embodiments, as Figure 8 As shown, the above step S203, based on the difference file, constructs the layer file of the new incremental layer, which specifically includes the following steps S801 to S803.

[0145] Step S801: Generate a deletion mark file corresponding to the deleted file.

[0146] Step S802 : Generate a layer file in a preset format based on the newly added files, modified files, and deleted marked files in accordance with the same directory structure as the base image.

[0147] Step S803: obtaining a layer file of a new incremental layer based on the layer file in a preset format.

[0148] A delete marker file is a file generated to record file deletion operations, such as a whiteout file (blank file / deletion marker). For example, if a file exists in the base layer but has been deleted from the mounted directory, a whiteout file needs to be generated in the corresponding path to record the deletion operation, in compliance with the OCI Whiteout specification.

[0149] It should be noted that in the OCI specification, if the base image layer has a file, and the container runtime "deletes" it through the writable layer (actually hides it in the joint mount), a special Whiteout file (named in the format of .wh. <filename>, empty files) are placed in the corresponding path to record this logical deletion operation. This ensures that after building a new incremental layer, it complies with the OCI image specification for layered file deletion, allowing the new image to correctly reproduce the effect of "deleting the original base layer files" when running in other environments.

[0150] The default format is the tar.gz format, which is a common storage format for OCI image layers.

[0151] The layer file in a preset format refers to a compressed package in a preset format obtained by adding new files, modifying files, and deleting marked files in the same directory structure as the base image, such as a gzip-compressed tarball.

[0152] The layer file of the new incremental layer refers to a layer file generated in a preset format.

[0153] Specifically, the server generates a deletion marker file corresponding to the deleted file according to the deletion marker file generation instruction; then obtains the same directory structure and preset format as the base image, and generates a layer file in the preset format based on the newly added files, modified files and deletion marker files according to the same directory structure as the base image, and finally uses the generated layer file in the preset format as the layer file of the new incremental layer.

[0154] For example, the server generates a Whiteout file in the corresponding path based on the files that exist in the base layer but have been deleted in the mounted directory. Then, the newly added / modified files and the Whiteout file are packaged into a gzip-compressed tarball according to the same directory structure as the base image as the layer file of the new incremental layer.

[0155] The technical solution provided by this embodiment generates a deletion marker file corresponding to the deleted file, and then generates a layer file in a preset format based on the newly added files, modified files and deletion marker file in accordance with the same directory structure as the basic image, and finally obtains the layer file of the new incremental layer based on the layer file in the preset format; in this way, when exporting the container image file, only the layer file of the new incremental layer is generated, and there is no need to export the layer file of each image layer, thereby reducing file transmission redundancy and helping to improve the export efficiency of the container image file.

[0156] In some embodiments, as Figure 9 As shown, the above step S204 writes the layer file of the incremental layer into the image layer file directory of the base image, and updates the manifest file and index file of the base image to obtain the image file of the target container, which specifically includes the following steps S901 to S902. Among them:

[0157] Step S901: determine the file identifier of the layer file of the incremental layer.

[0158] Step S902 : According to the file identifier of the layer file of the incremental layer, the layer file of the incremental layer is written into the location corresponding to the file identifier in the image layer file directory of the base image.

[0159] The file identifier of the incremental layer layer file refers to the unique identification information of the incremental layer layer file, specifically a hash value, such as a sha256 hash value. The file identifier of the incremental layer layer file is used as the file name of the incremental layer layer file.

[0160] Specifically, the server calculates the hash value (such as the sha256 hash value) of the layer file of the incremental layer through a hash algorithm (such as the sha256 algorithm), and then uses the hash value of the layer file of the incremental layer as the file identifier of the layer file of the incremental layer. Finally, according to the file identifier of the layer file of the incremental layer, the layer file of the incremental layer is written into the location corresponding to the file identifier in the image layer file directory of the base image, so that the layer file of the incremental layer is recorded in the image layer file directory of the base image.

[0161] For example, the server calculates the sha256 digest of the layer file of the new incremental layer and writes the layer file of the new incremental layer to blobs / sha256 / according to the sha256 digest of the layer file of the incremental layer (as the file name). <digest>in the directory.

[0162] The technical solution provided by this embodiment determines the file identifier of the layer file of the incremental layer, and then writes the layer file of the incremental layer to the location corresponding to the file identifier in the image layer file directory of the base image according to the file identifier of the layer file of the incremental layer; in this way, when exporting the container image file, only the layer file of the incremental layer is written to the location corresponding to the file identifier of the layer file of the incremental layer in the image layer file directory of the base image, and there is no need to export the layer file of each image layer, thereby reducing file transmission redundancy, which is beneficial to improving the export speed of the container image file, thereby further improving the export efficiency of the container image file.

[0163] In some embodiments, as Figure 10 As shown, the layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container, which specifically includes the following steps S1001 to S1005. Among them:

[0164] Step S1001: Obtain metadata of the layer file of the incremental layer.

[0165] Step S1002 : Add metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image to obtain an updated manifest file.

[0166] Step S1003: Obtain metadata of the updated manifest file.

[0167] Step S1004: Update the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain the updated index file.

[0168] Step S1005 : obtaining the image file of the target container based on the image layer file directory, the updated manifest file, and the updated index file of the layer file written into the incremental layer.

[0169] The metadata of the layer file of the incremental layer includes the media type, size, digest (ie, hash value), etc. of the layer file of the incremental layer.

[0170] The image layer array includes metadata of the layer files of each image layer of the base image.

[0171] Among them, the metadata of the layer file of the incremental layer is added to the end of the image layer array of the updated manifest file.

[0172] The metadata of the updated manifest file includes the name, size, digest (ie, hash value), etc. of the updated manifest file.

[0173] The metadata of the manifest file in the updated index file has been updated to the metadata of the updated manifest file.

[0174] Specifically, the server obtains the metadata of the layer file of the incremental layer, such as the media type, size, and digest (i.e., hash value), through the metadata acquisition instruction; then, the server identifies the manifest file of the base image, and adds the metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image to obtain the updated manifest file; then, through the metadata acquisition instruction, the server obtains the metadata of the updated manifest file, such as the size and digest (i.e., hash value), and updates the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain the updated index file; finally, based on the image layer file directory of the layer file written to the incremental layer, the updated manifest file, and the updated index file, the server obtains the image file of the target container.

[0175] For example, the server reads the storage directory of the manifest.json file (manifest file) blobs / sha256 / <base-manifest>, append the descriptor of the new incremental layer to the end of the layers array in the manifest.json file to update the manifest.json file; the descriptor refers to the metadata description, including fields such as mediaType, size, and digest; then recalculate the digest and size of the updated manifest.json file, replace the digest and size of the corresponding entries in the index.json file, and update the index.json file to ensure the consistency of the entire OCI layout.

[0176] The technical solution provided by this embodiment obtains the metadata of the layer file of the incremental layer, then adds the metadata of the layer file of the incremental layer to the end of the mirror layer array of the manifest file of the base image to obtain the updated manifest file, then obtains the metadata of the updated manifest file, and updates the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain the updated index file, and finally obtains the image file of the target container based on the mirror layer file directory, updated manifest file and updated index file of the layer file written to the incremental layer; in this way, the purpose of obtaining the image file of the target container based on the mirror layer file directory, updated manifest file and updated index file of the layer file written to the incremental layer is achieved, that is, only the layer file of the incremental layer is written, and the corresponding manifest file and index file are updated, without exporting the layer file of each mirror layer, and without exporting the entire container image file, thereby reducing file transmission redundancy, which is conducive to improving the export efficiency of the container image file, and avoiding the defect of exporting the entire container image file, which leads to low export efficiency of the container image file.

[0177] In some embodiments, the present application further provides an image file processing device, which may include a communication module and a processor; the processor is coupled to the communication module.

[0178] like Figure 2 As shown, the processor is configured as follows:

[0179] Step S201 : in response to an image file export request for a target container, obtain target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container.

[0180] Step S202 : Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file.

[0181] Step S203: constructing a new incremental layer file based on the difference file;

[0182] Step S204, write the layer file of the incremental layer into the image layer file directory of the base image, and update the manifest file and index file of the base image to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

[0183] The image file processing device may be a server.

[0184] Specifically, the communication module receives an image file export request for a target container sent by a terminal, and sends the image file export request for the target container to the processor. In response to the image file export request for the target container, the processor obtains the target file system directory information of the target container, then obtains the target file system directory information of the base image, and compares the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image, such as newly added files, modified files, and deleted files. Then, based on the difference file, a new incremental layer layer file is constructed. Finally, the incremental layer layer file is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container.

[0185] It should be noted that for the specific implementation process of the above-mentioned image file processing device, please refer to Figure 2 The related embodiments of the image file processing method shown are not described in detail here.

[0186] The technical solution provided by this embodiment only needs to construct a new incremental layer file based on the difference file of the target container relative to the base image when exporting the container image file, and write the incremental layer layer file into the image layer file directory of the base image, as well as update the manifest file and index file of the base image. There is no need to export the entire image file of the target container, that is, there is no need to copy the entire file, thereby simplifying the container image file export process, which is conducive to improving the export speed of the container image file, thereby improving the export efficiency of the container image file, and avoiding the defect that the export speed of the container image file is slow and the export efficiency of the container image file is low.

[0187] In some embodiments, the processor is further configured to:

[0188] Download the base image from the image repository; generate the target file system directory information of the base image according to the preset image layout format.

[0189] In some embodiments, the processor is further configured to:

[0190] Obtain identification information of the image layer of the base image in the image repository, and obtain identification information of the image layer stored in the local database; compare the identification information of the image layer of the base image in the image repository with the identification information of the image layer stored in the local database to obtain a comparison result; based on the comparison result, determine the image layer to be downloaded of the base image in the image repository; the image layer to be downloaded is used to represent the image layer contained in the base image in the image repository and not stored in the local database; download the image layer to be downloaded of the base image from the image repository.

[0191] In some embodiments, the processor is further configured to:

[0192] In response to an image file export request for a target container, a container identifier of the target container is obtained; a mount information file corresponding to the container identifier of the target container is obtained; file system layering information of an overlay layer of the target container is identified from the mount information file; file system directory information of a writable layer of the target container is identified from the file system layering information; and target file system directory information of the target container is obtained based on the file system directory information of the writable layer of the target container.

[0193] In some embodiments, the processor is further configured to:

[0194] The target file system directory information of the base image is decompressed into the first temporary directory information, and the target file system directory information of the target container is mounted into the second temporary directory information; the directory structure of the first temporary directory information and the second temporary directory information are the same; the first temporary directory information and the second temporary directory information are compared to obtain a difference file.

[0195] In some embodiments, the processor is further configured to:

[0196] Identify files that exist in the second temporary directory information but do not exist in the first temporary directory information as newly added files of the target container; identify files that have the same file path but different file content in the second temporary directory information and the first temporary directory information as modified files of the target container; identify files that exist in the first temporary directory information but do not exist in the second temporary directory information as deleted files of the target container; and obtain a difference file based on the newly added files, modified files, and deleted files.

[0197] In some embodiments, the processor is further configured to:

[0198] Generate a deletion marker file corresponding to the deleted file; generate a layer file in a preset format based on the newly added files, modified files and deletion marker file according to the same directory structure as the base image; obtain the layer file of the new incremental layer based on the layer file in the preset format.

[0199] In some embodiments, the processor is further configured to:

[0200] Determine the file identifier of the layer file of the incremental layer; and write the layer file of the incremental layer to a location corresponding to the file identifier in the image layer file directory of the base image according to the file identifier of the layer file of the incremental layer.

[0201] In some embodiments, the processor is further configured to:

[0202] Get the metadata of the layer file of the incremental layer; add the metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image to obtain the updated manifest file; get the metadata of the updated manifest file; update the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain the updated index file; based on the image layer file directory of the layer file written into the incremental layer, the updated manifest file and the updated index file, get the image file of the target container.

[0203] In some embodiments, as Figure 14 As shown, in order to more clearly describe the signaling interaction process between the modules of the image file processing device (i.e., the server), the present application also provides another image file processing method, which may include the following steps:

[0204] Step 1: The user or tool triggers the pull of the base image and generates the OCI layout.

[0205] Among them, through the CLI (Command Line Interface) such as ctr or podman pull, the image to be used as the baseline is pulled locally, and the directory structure such as oci-layout, index.json, blobs / sha256 is generated according to the OCI Image Layout (Open Container Initiative Image Layout) specification.

[0206] Step 2: The target container PID is obtained through user or tool triggering.

[0207] The PID of the target container is obtained through the crictl ps or ctr tasks ls command.

[0208] Step 3: Trigger parsing of / proc / by user or tool <pid> / mountinfo.

[0209] Among them, the tool reads / proc / <pid> / mountinfo, filters out Overlay type entries, and locates the upper and lower directory paths of the target container from the Overlay type entries, thereby determining the actual file system root path.

[0210] Step 4: The container runtime provides the mount point path to the difference extraction module.

[0211] When the container is running, it sends the parsed upper-level directory to the difference extraction module.

[0212] Step 5: The difference extraction module decompresses the base image and mounts the Overlay.

[0213] The difference extraction module bind-mounts the parsed upper directory to a local temporary path, such as / mnt / overlay-upper, for subsequent file comparison operations; then it decompresses all layers of the base image to another temporary directory (such as / mnt / image-base), maintaining the original directory tree structure.

[0214] Step 6: The difference extraction module generates an A (new) / C (modified) / D (deleted) list through file comparison commands such as diff and rsync.

[0215] The difference extraction module generates a list of difference files through file comparison commands such as diff and rsync. For files that exist in the base layer but have been deleted in the mount directory, a .wh. <filename>Empty file to record deletion operations, so as to obtain A (new) / C (modify) / D (delete) lists.

[0216] Step 7: The difference extraction module passes the difference file list to the incremental layer construction module.

[0217] After obtaining the difference file list, the difference extraction module sends the difference file list to the incremental layer construction module.

[0218] In step 8, the incremental layer construction module packages the incremental tarball (ie, compressed package) and calculates the digest.

[0219] The incremental layer construction module packages the newly added / modified files and the Whiteout files in the difference file list into a gzip-compressed tarball according to the same directory structure as the base image, calculates its sha256 digest (i.e., digest), and writes the packaged gzip-compressed tarball to blobs / sha256 / <digest>in the directory.

[0220] Step 9: The incremental layer builds the module to update manifest.json (manifest file) and index.json (index file).

[0221] Among them, the incremental layer construction module reads blobs / sha256 / <base-manifest>, append the new incremental layer descriptor (including mediaType, size, digest and other fields) to the end of the layers array in the manifest.json file to update the manifest.json file.

[0222] Among them, the incremental layer construction module recalculates the summary and size of the updated manifest.json file, and uses the summary and size of the updated manifest.json file to replace the summary and size of the corresponding entry in the index.json file to update the index.json file.

[0223] Step 10: The incremental layer construction module uploads the newly generated incremental image layout to the image registry.

[0224] Among them, the incremental layer construction module uploads the newly generated incremental image layout to the image registry (i.e., the image warehouse), and stores the newly generated incremental image layout through the image registry for subsequent pulling and deployment.

[0225] The technical solution provided by this embodiment only needs to construct a new incremental layer file based on the difference file of the target container relative to the base image when exporting the container image file, and write the incremental layer layer file into the image layer file directory of the base image, as well as update the manifest file and index file of the base image. There is no need to export the entire image file of the target container, that is, there is no need to copy the entire file, thereby simplifying the container image file export process, which is conducive to improving the export speed of the container image file, thereby improving the export efficiency of the container image file, and avoiding the defect that the export speed of the container image file is slow and the export efficiency of the container image file is low.

[0226] In some embodiments, in order to more clearly illustrate the image file processing method provided by the embodiments of the present application, the image file processing method is specifically described below using a specific embodiment. The present application also provides a method for quickly and efficiently exporting large-volume containers in AI scenarios into images, which involves container image generation and management technology, especially in AI application scenarios where the container carries model files resulting in the container being tens of GB in size. A general container image generation method based on container differential export and incremental layer construction. Specifically including the following contents:

[0227] In AI application scenarios, the need for efficient and secure container image export is particularly urgent, as shown in the following points: (1) Large model files lead to large image sizes: AI inference and training containers usually need to preload several GB or even tens of GB of model parameters and related dependencies. Common deep learning container images can be over 10 GB in size, which seriously affects the speed of pulling and starting images. (2) Export bottlenecks in the development-debugging-solidification process: AI developers are accustomed to performing model training and function debugging in containers. After completion, they need to export the container environment as an image to solidify the debugging status and push it to the production environment. When the image size is too large, each docker save (local image archiving) or docker push (remote image upload) operation will significantly slow down the delivery process due to the time-consuming network transmission and storage IO (Input / Output). (3) Removal of Docker Shim (Docker Adapter) and Security Risks: The traditional solution of exporting images by mounting the host's docker.sock (the local communication socket of the Docker engine) and calling the Docker API (the interface protocol for remotely controlling the Docker engine) is no longer available since Kubernetes v1.24 (container orchestration platform version 1.24) because the Docker Shim has been removed; and mounting the socket is equivalent to granting the Pod full control over the host container runtime, which poses a serious security risk. (4) Redundancy and low efficiency of single-layer image solutions: The method of using the client to directly read the container storage path to package and export can only generate a single layer containing all files, and cannot reuse the existing base image layer, resulting in the new image being almost a "full copy", further increasing the image size and network redundancy. (5) Multi-runtime compatibility requirements: In AI scenarios, there are various container runtimes (Docker, Containerd, CRI-O, Podman, etc.). The ideal solution must be compatible with all runtimes that comply with the OCI standard to ensure consistent operation in different clusters and cloud environments.

[0228] Furthermore, in AI application scenarios, developers often need to export the running container environment as an image to solidify debugging results and move them to the production environment. The following details several mainstream solutions, their corresponding drawbacks, and their causes.

[0229] 1. Mount the Docker Socket and call the Docker API to export.

[0230] Solution description: By mounting the host's / var / run / docker.sock in the Pod, use the Docker CLI or REST API (such as docker commit and docker save) to export snapshots and images of running containers.

[0231] Flaws and Causes: 1) High Security Risk: Mounting docker.sock grants the container full control over the host Docker daemon, allowing attackers to gain root privileges on the host and launch arbitrary operations against other containers or the host system. 2) Limited Compatibility: Kubernetes removed built-in support for the Docker Shim as of v1.24, making it impossible to directly rely on the Docker runtime. To continue using this solution, the Docker engine must be installed and maintained on all worker nodes, increasing the operational burden. 3) Operational Complexity: Opening docker.sock requires additional configuration in the Pod Security Policy (PSP / PSA), making it difficult to be compatible with strict cloud-native security pipelines.

[0232] 2. Use the Podman / CRI-O client to export the container storage path.

[0233] Solution description: Bypass Docker and use Podman, CRI-O, or containerd's CLI (such as podmanexport and ctr) to directly read the container's underlying storage (overlay2, fuse, etc.) and package the entire file system into a single tarball.

[0234] Defects and Causes: 1) Extensive Single-Layer Redundancy: The export result is typically a monolithic layer containing all file data, which cannot be incrementally reused with the original base image layer. This results in a large new image and the repeated transmission of large amounts of unchanged data. 2) Lack of Layered Optimization: The single-layer structure does not utilize the layered caching mechanism of OCI / Docker. Each update requires the entire layer to be retransmitted, severely impacting network bandwidth and distribution efficiency. 3) Incomplete Metadata: When directly exporting a file system, the layered descriptions in the image's manifest.json and config.json files often cannot be automatically updated. Manual modification or additional tool support is required, increasing secondary development costs.

[0235] 3. Export using the CRI tool (crictl) and the containerd CLI (ctr).

[0236] Solution description: Use crictl, the CRI debugging tool recommended by Kubernetes, or the ctrimage export command provided by containerd to export the image or runtime snapshot stored in containerd as a tarball.

[0237] Flaws and Causes: 1) Limited Functionality: crictl is only used for listing and debugging containers and does not support image import / export operations, making it incapable of directly meeting export requirements. 2) Dependency on containerd: The ctr command requires direct access to the containerd daemon on the node and uses a specific namespace. This is strictly dependent on the cluster runtime environment, making it difficult to use universally in Docker or other CRI-O environments. 3) Complex Operation: Requires logging into the node and executing multiple commands (such as export, copy, and load), making it unsuitable for quick, one-click triggering of export tasks in automated pipelines.

[0238] Based on the incremental export concept of the OCI layering mechanism, this application compares the baseline image with the file system snapshot of the runtime container, extracts only the changed files and generates a new incremental layer; then appends the layer descriptor (metadata) to the manifest.json (manifest file) of the original image, achieving minimal addition of image layers. This not only takes into account security and efficiency, but also fully complies with the OCI image specification and is adaptable to multiple runtimes. Specific technical points include the following:

[0239] 1. OCI layered incremental export mechanism: 1) By comparing the differences between the OverlayFS mount point at container runtime and the base image layer, an incremental layer (containing only modified / newly added files and whiteout tags) is generated, achieving layered reuse and image size optimization; 2) Unlike existing single-layer export solutions (such as Podman export), this mechanism retains the layered structure of OCI images.

[0240] 2. Runtime-independent container status capture method: 1) By parsing / proc / <pid>1) It uses Docker's . / mountinfo API to obtain the OverlayFS mount path, rather than relying on the API of a specific runtime (such as Docker / containerd); 2) It is compatible with multiple container runtimes (such as Docker / containerd / Podman / CRI-O).

[0241] 3. Automatic Whiteout processing mechanism: 1) Automatically detect and generate .wh. <filename>file, meeting the OCI image deletion operation specifications; 2) Different from the traditional method of manually modifying the image configuration file.

[0242] 4. Dynamic update algorithm for image metadata: 1) Automatically updates the hierarchical descriptions of manifest.json and index.json to ensure that the new image complies with the OCI specification; 2) Avoids the need to manually adjust the image configuration after export.

[0243] 5. Security isolation difference comparison method: 1) Bind-mount the container upperdir to a temporary directory for file comparison without directly accessing the runtime daemon.

[0244] Furthermore, the innovations of this application include the following:

[0245] 1. Security: No need to mount docker.sock, state capture is achieved through file system-level comparison; eliminating the risk of container escape and complying with the zero-trust security model.

[0246] 2. Layering mechanism: Generate incremental layers and merge them with the base image layer; retain the OCI layered structure and use the caching mechanism to reduce duplicate data transmission.

[0247] 3. Runtime dependency: Based on universal OverlayFS mount point parsing, it adapts to any runtime that complies with the OCI standard; it achieves cross-runtime compatibility and reduces operation and maintenance complexity.

[0248] 4. Image building efficiency: only package difference files (about 10-30% of the data volume); reduce more than 90% of network transmission and storage consumption.

[0249] 5. Metadata processing: Automatically update manifest.json and index.json to generate compliant OCI images; achieve end-to-end automation and avoid secondary development costs.

[0250] 6. Deletion operation support: Automatically generate Whiteout files to identify deletion operations; fully implement OCI specifications to ensure the accuracy of image building.

[0251] 7. Adaptability to deployment scenarios: Supports Kubernetes native environment (no node login required), can be integrated into CI / CD pipeline through Job / CronJob; adapts to cloud-native architecture and achieves seamless integration.

[0252] Specifically, refer to Figure 14 This application covers the complete process from pulling the base image, mounting the container file system, extracting the difference file to building and updating the OCI incremental image layout. It is compatible with multiple container runtimes, does not require Docker Shim, and fully utilizes the OCI layering mechanism to achieve efficient and secure incremental image export. The interaction process between each component is as follows Figure 14 The specific contents are as follows:

[0253] 1. Pull the base image and prepare the OCI layout.

[0254] 1) Pull the base image: Use the CLI (ctr or podman pull) to pull the image to be used as the baseline to the local computer, and generate the oci-layout, index.json, blobs / sha256, and other directory structures according to the OCI Image Layout specification.

[0255] 2) Verification and caching: Verify the downloaded layer hash. If the same layer is already available locally, skip downloading it. Use the OCI layered caching mechanism to reduce redundant transmission.

[0256] 2. Mount the container root file system and obtain the diff list.

[0257] 1) Identify the runtime process: Get the PID of the target container through crictl ps or ctr tasks ls.

[0258] 2) Parse the Overlay mount point: read / proc / <pid> / mountinfo, filters Overlay type entries, locates the upper and lower directory paths of the container, and determines the actual file system root path.

[0259] 3) Mount to a temporary directory: Bind and mount the parsed upper directory to a local temporary path, such as / mnt / overlay-upper, for subsequent file comparison operations.

[0260] 3. Differential file extraction and Whiteout processing.

[0261] 1) Decompress the base image: Decompress all layers of the base image separately to another temporary directory (such as / mnt / image-base), maintaining the original directory tree structure.

[0262] 2) File comparison: Use one of the following three methods to obtain a list of differences:

[0263] diff-rq / mnt / image-base / mnt / overlay-upper;

[0264] rsync --dry-run --itemize-changes mode generates lists of newly added (A) and modified (C) files;

[0265] Directly tar --diff -f compares directory differences and outputs file paths.

[0266] 3) Whiteout file generation: For files that exist in the base layer but have been deleted in the mount directory, generate .wh in the corresponding path. <filename>Empty file to record deletion operations in compliance with the OCI whiteout specification.

[0267] 4. Incremental layer construction and metadata update.

[0268] 1) Package incremental layer: Package the newly added / modified files and whiteout files into a gzip compressed tarball according to the same directory structure as the base image, calculate its sha256 digest and write it to blobs / sha256 / <digest>Table of contents.

[0269] 2) Update manifest.json: read blobs / sha256 / <base-manifest>, append the new layer's descriptor (including mediaType, size, digest, and other fields) to the end of the layers array in the manifest.json file.

[0270] 3) Update index.json: Recalculate the digest and size of the updated manifest, and replace the digest and size of the corresponding entries in index.json to ensure the consistency of the entire OCI layout.

[0271] The technical solution provided in this embodiment can achieve the following technical effects: 1) High security: no host docker.sock is required, avoiding security risks caused by the abuse of Docker API; 2) Layered reuse: only the differential part is exported to generate an incremental layer, which is layered and merged with the original image to reduce redundant data; 3) Runtime independence: adaptable to multiple container runtimes such as Docker, containerd, CRI-O, Podman, etc.; 4) Small image size: based on differential packaging, the image size is significantly reduced, and the distribution and startup efficiency is improved.

[0272] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0273] Based on the same inventive concept, embodiments of the present application also provide an image file processing device for implementing the aforementioned image file processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the image file processing device can be found in the above-mentioned limitations of the image file processing method and will not be further elaborated here.

[0274] In an exemplary embodiment, Figure 15 As shown, an image file processing device 1500 is provided. The device can be applied to an image file processing device. The image file processing device may include a communication module and a processor, and the processor is coupled to the communication module. The device may include:

[0275] The information acquisition module 1510 is used to obtain the target file system directory information of the target container in response to the image file export request for the target container; the target file system directory information of the target container is used to represent the file system directory information of the writable layer of the target container.

[0276] The information comparison module 1520 is used to compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file.

[0277] The file construction module 1530 is used to construct a layer file of a new incremental layer based on the difference file.

[0278] The file processing module 1540 is used to write the layer file of the incremental layer into the image layer file directory of the base image, and update the manifest file and index file of the base image to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

[0279] Each module in the above-mentioned image file processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0280] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0281] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0282] It should be noted that the software or tools involved in this application are only used as an example. This application does not limit them, nor does it mean that the software or tools will be used in the specific implementation process.

[0283] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0284] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0285] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0286] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims. < / digest> < / filename> < / pid> < / filename> < / pid> < / digest> < / filename> < / pid> < / pid> < / digest> < / filename> < / pid> < / digest>

Claims

1. A method for processing an image file, characterized in that: The method comprises: In response to an image file export request for a target container, obtaining target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container; Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file; Building a new incremental layer file based on the difference file; The layer file of the incremental layer is written into the image layer file directory of the base image, and the manifest file and index file of the base image are updated to obtain the image file of the target container; the manifest file includes the metadata of each image layer in the base image, and the index file includes the metadata of the manifest file.

2. The method according to claim 1, characterized in that Before obtaining target file system directory information of the target container in response to the image file export request for the target container, the method further includes: Download the base image from the image repository; Generate target file system directory information of the base image according to a preset image layout format.

3. The method according to claim 2, characterized in that Downloading the base image from the image repository includes: Obtain the identification information of the image layer of the base image in the image repository, and obtain the identification information of the image layer stored in the local database; Compare the identification information of the image layer of the base image in the image repository with the identification information of the image layer stored in the local database to obtain a comparison result; Based on the comparison result, determining the image layer to be downloaded of the base image in the image repository; the image layer to be downloaded is used to represent the image layer contained in the base image in the image repository and not stored in the local database; Download the image layer to be downloaded of the base image from the image warehouse.

4. The method according to claim 1, wherein The step of obtaining target file system directory information of the target container in response to the image file export request for the target container includes: In response to an image file export request for a target container, obtaining a container identifier of the target container; Obtaining the mount information file corresponding to the container identifier of the target container; Identifying file system layering information of an overlay layer of the target container from the mount information file; Identifying file system directory information of a writable layer of the target container from the file system layer information; Based on the file system directory information of the writable layer of the target container, target file system directory information of the target container is obtained.

5. The method according to claim 1, wherein The comparing the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image includes: Decompressing the target file system directory information of the base image into the first temporary directory information, and mounting the target file system directory information of the target container into the second temporary directory information; the directory structure of the first temporary directory information and the second temporary directory information is the same; The first temporary directory information and the second temporary directory information are compared to obtain the difference file.

6. The method according to claim 5, characterized in that The comparing the first temporary directory information and the second temporary directory information to obtain the difference file includes: identifying a file that exists in the second temporary directory information and does not exist in the first temporary directory information as a new file in the target container; identifying a file having the same file path and different file content in the second temporary directory information and the first temporary directory information as a modified file of the target container; identifying a file that exists in the first temporary directory information and does not exist in the second temporary directory information as a deleted file of the target container; The difference file is obtained based on the newly added file, the modified file and the deleted file.

7. The method according to claim 6, characterized in that The step of constructing a new incremental layer file based on the difference file includes: Generate a deletion mark file corresponding to the deletion file; Generate a layer file in a preset format based on the newly added file, the modified file, and the deletion mark file according to the same directory structure as the base image; Based on the layer file in the preset format, a layer file of a new incremental layer is obtained.

8. The method according to any one of claims 1 to 7, characterized in that The step of writing the layer file of the incremental layer into the image layer file directory of the base image and updating the manifest file and index file of the base image to obtain the image file of the target container includes: Determine the file identifier of the layer file of the incremental layer; According to the file identifier of the layer file of the incremental layer, the layer file of the incremental layer is written into the location corresponding to the file identifier in the image layer file directory of the base image.

9. The method according to any one of claims 1 to 7, characterized in that The step of writing the layer file of the incremental layer into the image layer file directory of the base image and updating the manifest file and index file of the base image to obtain the image file of the target container includes: Obtain metadata of the layer file of the incremental layer; Add the metadata of the layer file of the incremental layer to the end of the image layer array of the manifest file of the base image to obtain an updated manifest file; Obtaining metadata of the updated manifest file; Updating the metadata of the manifest file in the index file to the metadata of the updated manifest file to obtain an updated index file; The image file of the target container is obtained based on the mirror layer file directory of the layer file written into the incremental layer, the updated manifest file, and the updated index file.

10. An image file processing device, characterized in that: include: Communication module; a processor coupled to the communication module and configured to: In response to an image file export request for a target container, obtaining target file system directory information of the target container; the target file system directory information of the target container is used to represent file system directory information of a writable layer of the target container; Compare the target file system directory information of the base image with the target file system directory information of the target container to obtain a difference file of the target container relative to the base image; the difference file includes at least one of a newly added file, a modified file, and a deleted file; Building a new incremental layer file based on the difference file; Write the layer file of the incremental layer into the image layer file directory of the base image, and update the manifest file and index file of the base image to obtain the image file of the target container; The manifest file includes metadata of each image layer in the base image, and the index file includes metadata of the manifest file.

Images